Statistically significant forecasting improvements: how much out-of-sample data is likely necessary?

Abstract

Testing the out-of-sample forecasting superiority of one model over another requires an a priori partitioning of the data into a model specification/estimation ('training') period and a model comparison/evaluation ('out-of-sample' or 'validation') period. How large a validation period is necessary for a given mean square forecasting error (MSFE) improvement to be statistically significant at the 5% level? If the forecast errors from each model are NIID and these errors are independent of one another, then the 5% critical points for the F distribution provide the answer to this question. But even optimal forecast errors from well-specified models can be serially correlated. And forecast errors are typically substantially crosscorrelated. For such errors, a validation period in excess of 100 observations long is typically necessary in order for a 20% MSFE reduction to be statistically significant at the 5% level. Illustrative applications using actual economic data are given.