When a good fit can be bad
Trends in Cognitive SciencesPublished 1 October 2002
Mark A. Pitt, In Jae Myung
Citations397
SJR quartileQ1
SJR score4.51
SNIP5.51
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
How should we select among computational models of cognition? Although it is commonplace to measure how well each model fits the data, this is insufficient. Good fits can be misleading because they can result from properties of the model that have nothing to do with it being a close approximation to the cognitive process of interest (e.g. overfitting). Selection methods are introduced that factor in these properties when measuring fit. Their success in outperforming standard goodness-of-fit measures stems from a focus on measuring the generalizability of a model's data-fitting abilities, which should be the goal of model selection.
Keywords
Computer Science
Journal of Wildlife ManagementModel Selection and Multimodel Inference: A Practical Information-Theoretic Approach
42,130 Citations2003Fred S. Guthery, Kenneth P. Burnham +1 more
The second edition of this book is unique in that it focuses on methods for making formal statistical inference from all the models in an a priori set (Multi-Model Inference).
Choice Reviews OnlineNeural networks for pattern recognition
18,687 Citations1994
This is the first comprehensive treatment of feed-forward neural networks from the perspective of statistical pattern recognition, and is designed as a text, with over 100 exercises, to benefit anyone involved in the fields of neural computation and pattern recognition.
Springer series in statisticsInformation Theory and an Extension of the Maximum Likelihood Principle
17,886 Citations1998H. Akaike
Texts in computer scienceAn Introduction to Kolmogorov Complexity and Its Applications
3,734 Citations2019Ming Li, Paul Vitányi
Perception & PsychophysicsThe psychometric function: I. Fitting, sampling, and goodness of fit
2,430 Citations2001Felix A. Wichmann, N. Jeremy Hill
Journal of the American Statistical AssociationBayes Factors
2,303 Citations1995Robert E. Kass, Adrian E. Raftery
Journal of Experimental Psychology GeneralAttention, similarity, and the identification-categorization relationship.
2,225 Citations1986Robert M. Nosofsky
A unified quantitative approach to modeling subjects' identification and categorization of multidimensional perceptual stimuli is proposed and tested and some support was gained for the hypothesis that subjects distribute attention among component dimensions so as to optimize categorization performance.
IEEE Transactions on Information TheoryFisher information and stochastic complexity
841 Citations1996J. Rissanen
A sharper code length is obtained as the stochastic complexity and the associated universal process are derived for a class of parametric processes by taking into account the Fisher information and removing an inherent redundancy in earlier two-part codes.
Psychological ReviewHow persuasive is a good fit? A comment on theory testing.
834 Citations2000Seth Roberts, Harold Pashler
A better way to test a theory with free parameters is to determine how the theory constrains possible outcomes, assess how firmly actual outcomes agree with those constraints, and determine if plausible alternative outcomes would have been inconsistent with the theory, allowing for the variability of the data.
Journal of the American Statistical AssociationModel Selection and the Principle of Minimum Description Length
804 Citations2001Mark Hansen, Bin Yu
This article reviews the principle of minimum description length (MDL) for problems of model selection, and illustrates the MDL principle by considering problems in regression, nonparametric curve estimation, cluster analysis, and time series analysis.
Psychological ReviewToward a method of selecting among computational models of cognition.
499 Citations2002Mark A. Pitt, In Jae Myung +1 more
A method of selecting among mathematical models of cognition known as minimum description length is introduced, which provides an intuitive and theoretically well-grounded understanding of why one model should be chosen.
Psychonomic Bulletin & ReviewApplying Occam’s razor in modeling cognition: A Bayesian approach
436 Citations1997In Jae Myung, Mark A. Pitt
This paper introduces a Bayesian model selection approach that formalizes Occam's razor, choosing the simplest model that describes the data well, and can be applied to the comparison of non-nested models as well as nested ones.
Psychological ReviewIntegration of featural information in speech perception.
347 Citations1978Gregg C. Oden, Dominic W. Massaro
A model for the identification of speech sounds is proposed that assumes that the acoustic cues are perceived independently, and provides a good description of the data, including these boundary changes, while still maintaining complete noninteraction at the feature evaluation stage of processing.
Journal of Experimental Psychology Human Perception & PerformanceModels of visual word recognition: Sampling the state of the art.
297 Citations1994Arthur M. Jacobs, Jonathan Grainger
Journal of Experimental Psychology Learning Memory and CognitionThirty categorization results in search of a model.
206 Citations2000J. David Smith, John Paul Minda
Proceedings of the National Academy of SciencesCounting probability distributions: Differential geometry and model selection
177 Citations2000In Jae Myung, Vijay Balasubramanian +1 more
This work argues that assessing the "complexity" of explanations is essential to a theoretically well-founded model selection procedure, and forms model complexity in terms of the geometry of the space of probability distributions, which provides a clear intuitive understanding of several extant notions of model complexity.
Journal of Mathematical PsychologyGUEST EDITORS' INTRODUCTION
82 Citations2000In Jae Myung, Marc R. Forster +1 more
Psychonomic Bulletin & ReviewFlexibility versus generalizability in model selection
55 Citations2003Mark A. Pitt, Woojae Kim +1 more
It is demonstrated that the results from model recovery simulations can be misleading if they are not interpreted relative to the data on which they were evaluated, and that BMS is a more robust selection method.
Journal of Experimental Psychology GeneralUsing parameter sensitivity and interdependence to predict model scope and falsifiability.
25 Citations1996Shu Li, Stephan Lewandowsky +1 more