Methods and Criteria for Model Selection
Journal of the American Statistical AssociationPublished 1 March 2004Open access
Joseph B. Kadane, Nicole A. Lazar
Citations443
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Model selection is an important part of any statistical analysis, and indeed is central to the pursuit of science in general. Many authors have examined this question, from both frequentist and Bayesian perspectives, and many tools for selecting the ``best model'' have been suggested in the literature. This paper considers the various proposals from a Bayesian decision-theoretic perspective.
Keywords
MathematicsDecision Sciences
Journal of the Royal Statistical Society Series B (Statistical Methodology)Regression Shrinkage and Selection Via the Lasso
51,790 Citations1996Robert Tibshirani
A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
TechnometricsApplied Regression Analysis
18,042 Citations2005
This tutorial discusses simple and multiple linear regression, diagnostics, model selection, models with categorical variables, and nonlinear models; logistic regression.
Springer series in statisticsInformation Theory and an Extension of the Maximum Likelihood Principle
17,886 Citations1998H. Akaike
The Annals of StatisticsBootstrap Methods: Another Look at the Jackknife
17,446 Citations1979B. Efron
Journal of the Royal Statistical Society Series B (Statistical Methodology)Cross-Validatory Choice and Assessment of Statistical Predictions
10,527 Citations1974M. Stone
Society for Industrial and Applied Mathematics eBooksThe Jackknife, the Bootstrap and Other Resampling Plans
7,903 Citations1982Bradley Efron
BiometrikaIdeal spatial adaptation by wavelet shrinkage
7,813 Citations1994David L. Donoho, Iain M. Johnstone
A new principle for spatially-adaptive estimation: selective wavelet reconstruction with an oracle inequality is described and a practical spatially adaptive method, RiskShrink, which works by shrinkage of empirical wavelet coefficients is developed.
BiometrikaReversible jump Markov chain Monte Carlo computation and Bayesian model determination
5,929 Citations1995Peter J. Green
A new framework for the construction of reversible Markov chain samplers that jump between parameter subspaces of differing dimensionality is proposed, which is flexible and entirely constructive, and should have wide applicability in model determination problems.
Neural ComputationBayesian Interpolation
4,377 Citations1992David Mackay
The Bayesian approach to regularization and model-comparison is demonstrated by studying the inference problem of interpolating noisy data by examining the posterior probability distribution of regularizing constants and noise levels.
Journal of the American Statistical AssociationBayesian Inference in Statistical Analysis.
3,873 Citations1975Joseph B. Kadane, George E. P. Box +1 more
Journal of the American Statistical AssociationApplied Regression Analysis (2nd ed).
3,064 Citations1981José Ferreira de Carvalho, Norman R. Draper +1 more
This book brings together a number of procedures developed for regression problems in current use and includes material that either has not previously appeared in a textbook or if it has appeared is not generally available.
Journal of the American Statistical AssociationVariable Selection via Gibbs Sampling
2,683 Citations1993Edward I. George, Robert E. McCulloch
Journal of the American Statistical AssociationComputing Bayes Factors by Combining Simulation and Asymptotic Approximations
1,967 Citations1997Thomas J. DiCiccio, Robert E. Kass +2 more
It is found that a simulated version of Laplace's method, with local volume correction, furnishes an accurate approximation that is especially useful when likelihood function evaluations are costly.
Journal of the Royal Statistical Society Series B (Statistical Methodology)On Bayesian Analysis of Mixtures with an Unknown Number of Components (with discussion)
1,913 Citations1997Sylvia Richardson, Peter J. Green
TechnometricsGraphical Models in Applied Multivariate Statistics
1,827 Citations1991Colin Goodall, Joe Whittaker
Journal of the American Statistical AssociationMarginal Likelihood from the Gibbs Output
1,812 Citations1995Siddhartha Chib
This work exploits the fact that the marginal density can be expressed as the prior times the likelihood function over the posterior density, so that Bayes factors for model comparisons can be routinely computed as a by-product of the simulation.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Assessment and Propagation of Model Uncertainty
1,592 Citations1995David Draper
A Bayesian approach to solving this problem that has long been available in principle but is only now becoming routinely feasible, by virtue of recent computational advances, is discussed and its implementation is examined in examples that involve forecasting the price of oil and estimating the chance of catastrophic failure of the U.S. Space Shuttle.
Journal of the American Statistical AssociationBayesian Variable Selection in Linear Regression
1,416 Citations1988Toby J. Mitchell, John J. Beauchamp
TechnometricsThe Relationship Between Variable Selection and Data Agumentation and a Method for Prediction
1,361 Citations1974David M. Allen
It is shown that data augmentation provides a rather general formulation for the study of biased prediction techniques using multiple linear regression and a way to obtain predictors given a credible criterion of good prediction is proposed.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Bayesian Model Choice: Asymptotics and Exact Calculations
1,239 Citations1994Alan E. Gelfand, Dipak K. Dey
A general predictive density is presented which includes all proposed Bayesian approaches the authors are aware of and using Laplace approximations they can conveniently assess and compare asymptotic behavior of these approaches.
Journal of the American Statistical AssociationModel Selection and Accounting for Model Uncertainty in Graphical Models Using Occam's Window
1,234 Citations1994David Madigan, Adrian E. Raftery
APPROACHES FOR BAYESIAN VARIABLE SELECTION
1,152 Citations1997Edward I. George, Robert E. McCulloch
Various hierarchical mixture prior formulations of variable selection uncertainty in normal linear regression models are described and compared, including the nonconjugate SSVS formulation of George and McCulloch (1993), as well as conjugate formulations which allow for analytical simplification.
International Statistical ReviewBayesian Graphical Models for Discrete Data
1,115 Citations1995David Madigan, Jeremy York +1 more
Journal of the Royal Statistical Society Series B (Statistical Methodology)Bayesian Model Choice Via Markov Chain Monte Carlo Methods
1,017 Citations1995Bradley P. Carlin, Siddhartha Chib
This paper presents a framework for Bayesian model choice, along with an MCMC algorithm that does not suffer from convergence difficulties, and applies equally well to problems where only one model is contemplated but its proper size is not known at the outset.
Statistical ScienceSimulating normalizing constants: from importance sampling to bridge sampling to path sampling
974 Citations1998Andrew Gelman, Xiao‐Li Meng
It is shown that the acceptance ratio method and thermodynamic integration are natural generalizations of importance sampling, which is most familiar to statistical audiences.
Journal of the American Statistical AssociationThe Intrinsic Bayes Factor for Model Selection and Prediction
960 Citations1996James O. Berger, Luis R. Pericchi
This article introduces a new criterion called the intrinsic Bayes factor, which is fully automatic in the sense of requiring only standard noninformative priors for its computation and yet seems to correspond to very reasonable actual Bayes factors.
On Bayesian Analysis of Mixtures with an Unknown Number of Components
937 Citations1997Sylvia Richardson, Peter J. Green
Journal of the Royal Statistical Society Series A (General)Sampling and Bayes' Inference in Scientific Modelling and Robustness
936 Citations1980George E. P. Box
Predictive checking functions for transformation, serial correlation, bad values, and their relation with Bayesian options are considered, and robustness is seen from a Bayesian viewpoint and examples are given.
Journal of the American Statistical AssociationA Predictive Approach to Model Selection
919 Citations1979Seymour Geisser, William F. Eddy
BiometrikaModel choice: a minimum posterior predictive loss approach
730 Citations1998Alan E. Gelfand
A predictive criterion where the goal is good prediction of a replicate of the observed data but tempered by fidelity to the observed values is proposed, which is obtained by minimising posterior loss for a given model.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Fractional Bayes Factors for Model Comparison
687 Citations1995Anthony O’Hagan
Property of partial Bayes factors are discussed, particularly in the context of weak prior information, and they are found to have advantages over other proposed methods of model comparison.
SIMULATING RATIOS OF NORMALIZING CONSTANTS VIA A SIMPLE IDENTITY: A THEORETICAL EXPLORATION
646 Citations1996Xiao‐Li Meng, Wing Hung Wong
A theoretical study of the usefulness of the following simple identity, with focus on (asymptotically) optimal and practical choices of α, and demonstrates that with sensible (not necessarily optimal) choices of β, the simulation error can be reduced by orders of magnitude.
The Annals of StatisticsThe Risk Inflation Criterion for Multiple Regression
531 Citations1994Dean P. Foster, Edward I. George
BiometrikaCalibration and empirical Bayes variable selection
528 Citations2000Edward I. George
Empirical Bayes selection criteria that use hyperparameter estimates instead of fixed choices are proposed and approximate adaptively the performance of the best fixed-penalty criterion across a variety of orthogonal and nonorthogonal set-ups, including wavelet regression.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Multivariate Bayesian Variable Selection and Prediction
378 Citations1998Philip J. Brown, Marina Vannucci +1 more
The marginal posterior distribution of the binary latent vector of the multivariate regression model with p regressors is derived and the approach illustrated on compositional analysis of data involving three sugars with 160 near infrared absorbances as regressors.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Posterior Bayes Factors
363 Citations1991Murray Aitkin
Journal of the Royal Statistical Society Series B (Statistical Methodology)Predictive Model Selection
358 Citations1995Purushottam W. Laud, Joseph G. Ibrahim
Journal of the Royal Statistical Society Series B (Statistical Methodology)Bayes Factors and Choice Criteria for Linear Models
342 Citations1980A. F. M. Smith, David Spiegelhalter
Journal of the Royal Statistical Society Series D (The Statistician)Experiences in elicitation [Read before The Royal Statistical Society at a meeting on 'Elicitation' on Wednesday, April 16th, 1997, the President, Professor A. F. M. Smith in the Chair]
335 Citations1998Joseph B. Kadane, Lara J. Wolfson
The psychology of elicitation and the currently available methods are briefly reviewed, but the primary discussion is on the distinction between 'general' elicitation methods for a class of problems and 'application-specific' methods which are useful only once.
Journal of the Royal Statistical Society Series D (The Statistician)Eliciting expert beliefs in substantial practical applications [Read before The Royal Statistical Society at ameeting on 'Elicitation' on Wednesday, april 16th, 1997, the President, Professor A. F. M. Smithin the Chair]
310 Citations1998Anthony O’Hagan
The practical elicitation of expert beliefs is considered through two contrasting examples, and a common principle of trying to identify and elicit separately the various sources of expert uncertainty is identified.
Journal of the American Statistical AssociationInteractive Elicitation of Opinion for a Normal Linear Model
292 Citations1980Joseph B. Kadane, James Dickey +3 more
Journal of the American Statistical AssociationMarkov Chain Monte Carlo Methods for Computing Bayes Factors
280 Citations2001Cong Han, Bradley P. Carlin
It is found that the joint model-parameter space search methods perform adequately but can be difficult to program and tune, whereas the marginal likelihood methods often are less troublesome and require less additional coding.
Journal of the American Statistical AssociationThe Little Bootstrap and other Methods for Dimensionality Selection in Regression: X-Fixed Prediction Error
272 Citations1992Leo Breiman
Journal of EconometricsEstimating regression models of finite but unknown order
240 Citations1981John Geweke, Richard Meese
Journal of the Royal Statistical Society Series B (Statistical Methodology)The Choice of Variables in Multiple Regression
236 Citations1968D. V. Lindley
BiometricsEfficient Screening of Nonnormal Regression Models
235 Citations1978Jerald F. Lawless, K. Singhal
Journal of the Royal Statistical Society Series C (Applied Statistics)A Comparison of the Akaike and Schwarz Criteria for Selecting Model Order
178 Citations1988Anne B. Koehler, Emily S. Murphree
TechnometricsOn Some Criteria for Estimating the Order of a Markov Chain
177 Citations1981Richard W. Katz
International Statistical ReviewBayesian Hypothesis Testing: a Reference Approach
133 Citations2002José M. Bernardo, Raúl Rueda
Journal of the Royal Statistical Society Series B (Statistical Methodology)The Covariance Inflation Criterion for Adaptive Model Selection
120 Citations1999Robert Tibshirani, Keith Knight
Canadian Journal of StatisticsInference for nonconjugate Bayesian Models using the Gibbs sampler
116 Citations1991Bradley P. Carlin, Nicholas G. Polson
The Gibbs sampler technique is proposed as a mechanism for implementing a conceptually and computationally simple solution in such a framework and the result is a general strategy for obtaining marginal posterior densities under changing specification of the model error densities and related prior densities.
TestProperties of intrinsic and fractional Bayes factors
99 Citations1997Anthony O’Hagan
This paper identifies and contrasts various properties of these methods, with particular reference to coherence and practicality, which have been proposed for Bayesian model comparison when prior information about model parameters is weak.
Journal of the Royal Statistical Society Series B (Statistical Methodology)A Predictive Model Selection Criterion
88 Citations1984Antonella Martini, Fulvio Spezzaferri
Computational Statistics & Data AnalysisBayesian model choice based on Monte Carlo estimates of posterior model probabilities
78 Citations2004Peter Congdon
An approach is outlined here that produces posterior model probabilities and hence Bayes factor estimates but not marginal likelihoods and uses a Monte Carlo approximation based on independent MCMC sampling of two or more different models.
BiometrikaThe choice of variables in multivariate regression: a non-conjugate Bayesian decision theory approach
75 Citations1999Philip J. Brown
ACCURATE AND STABLE BAYESIAN MODEL SELECTION: THE MEDIAN INTRINSIC BAYES FACTOR*
67 Citations2002Luis R. Pericchi
An implementation of the IBF strategy called the Median IBF is studied, which seems to be a simple and very generally applicable IBF, which works well for nested or non-nested models, and even for small or moderate sample sizes; some of these situations can cause difficulties for other versions of IBF.
Journal of the American Statistical AssociationInteractive Elicitation of Opinion for a Normal Linear Model
58 Citations1980Joseph B. Kadane, James Dickey +3 more
The Annals of StatisticsElicitation of Prior Distributions for Variable-Selection Problems in Regression
53 Citations1992Paul H. Garthwaite, James Dickey
RePEc: Research Papers in EconomicsBayesian Decision Theory and the Simplification of Models
30 Citations1980Joseph B. Kadane, James Dickey
Journal of the Royal Statistical Society Series B (Statistical Methodology)Non-conjugate Prior Distribution Assessment for Multivariate Normal Sampling
25 Citations2001Paul H. Garthwaite, Shafeeqah A. Al-Awadhi
Journal of Statistical Planning and InferenceSome comments on Bayes factors
19 Citations1997D. V. Lindley
Wiley StatsRef: Statistics Reference Online<scp>B</scp>ayesian Graphical Models
12 Citations2016Finn V. Jensen, Thomas D. Nielsen
For the sake of simplicity this article focuses on models with only discrete variables, and refers to the variable values as states.
…