Evaluating Automatic Model Selection

Abstract

We outline a range of criteria for evaluating model selection approaches that have been used in the literature. Focusing on three key criteria, we evaluate automatically selecting the relevant variables in an econometric model from a large candidate set. General-tospecific selection is outlined for a regression model in orthogonal variables, where only one decision is required to select, irrespective of the number of regressors. Comparisons with an automated model selection algorithm, Autometrics (Doornik, 2009), show similar properties, but not restricted to orthogonal cases. Monte Carlo experiments examine the roles of post-selection bias corrections and diagnostic testing, and evaluate selection in dynamic models by costs of search versus costs of inference.