Complexity Regularization with Application to Artificial Neural Networks
Published 1 January 1991
Andrew R. Barron
Citations160
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper defines general complexity regularization criteria and establishes bounds on the statistical risk of the estimated functions and establishes consistency, yield rates of convergence, and the near asymptotic optimality of the model selection criterion in both parametric and nonparametric cases.
Abstract
Concepts of universal data compression lead to minimum description-length criteria for parsimonious statistical model selection for the estimation of functions. In this paper we define general complexity regularization criteria and establish bounds on the statistical risk of the estimated functions. These bounds establish consistency, yield rates of convergence, and demonstrate the near asymptotic optimality of the model selection criterion in both parametric and nonparametric cases. A fundamental role is played by an index of resolvability that quantifies the tradeoff between complexity and accuracy of candidate models. Applications are given to polynomial regression and artificial neural networks.
Keywords
Computer ScienceEngineering
The MIT Press eBooksParallel Distributed Processing
15,227 Citations1986David E. Rumelhart, James L. McClelland +1 more
Mathematics of Control Signals and SystemsApproximation by superpositions of a sigmoidal function
13,684 Citations1989George Cybenko
The Annals of StatisticsMultivariate Adaptive Regression Splines
8,084 Citations1991Jerome H. Friedman
A new method is presented for flexible regression modeling of high dimensional data that takes the form of an expansion in product spline basis functions, where the number of basis functions as well as the parameters associated with each one are automatically determined by the data.
IEEE ASSP MagazineAn introduction to computing with neural nets
7,858 Citations1987Richard P. Lippmann
Springer series in statisticsProbability Inequalities for sums of Bounded Random Variables
6,947 Citations1994Wassily Hoeffding
Journal of the American Statistical AssociationProbability Inequalities for Sums of Bounded Random Variables
4,750 Citations1963Wassily Hoeffding
Information science and statisticsEstimation of Dependences Based on Empirical Data
2,238 Citations2006Vladimir Vapnik
The Annals of StatisticsA Universal Prior for Integers and Estimation by Minimum Description Length
1,667 Citations1983J. Rissanen
Estimation based upon the principle of minimizing the total number of binary digits required to rewrite the observed data, when each observation is given with some precision is studied.
IEEE Transactions on Information TheoryUniversal coding, information, prediction, and estimation
903 Citations1984J. Rissanen
A connection between universal codes and the problems of prediction and statistical estimation is established, and a known lower bound for the mean length of universal codes is sharpened and generalized, and optimum universal codes constructed.
Journal of the American Statistical AssociationProbability Inequalities for the Sum of Independent Random Variables
756 Citations1962G. G. Bennett
Neural NetworksConnectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings
678 Citations1990Halbert White
It is shown that sufficiently complex multilayer feedforward networks are capable of representing arbitrarily accurate approximations to arbitrary mappings by proving the consistency of a class of connectionist nonparametric regression estimators for arbitrary (square integrable) regression functions.
Medical Entomology and ZoologySelf-Organizing Methods in Modeling: Gmdh Type Algorithms
594 Citations1984Stanley J. Farlow
The Annals of StatisticsAsymptotic Optimality for $C_p, C_L$, Cross-Validation and Generalized Cross-Validation: Discrete Index Set
521 Citations1987Ker-Chau Li
IEEE Transactions on Information TheoryMinimum complexity density estimation
497 Citations1991Andrew R. Barron, Thomas M. Cover
An index of resolvability is proved to bound the rate of convergence of minimum complexity density estimators as well as the information-theoretic redundancy of the corresponding total description length to demonstrate the statistical effectiveness of the minimum description-length principle as a method of inference.
The Annals of StatisticsA Simple Lemma on Greedy Approximation in Hilbert Space and Convergence Rates for Projection Pursuit Regression and Neural Network Training
474 Citations1992Lee Jones
A general convergence criterion for certain iterative sequences in Hilbert space is presented and results establish an 0(1/ F4n) nonsampling convergence rate for projection pursuit regression and neural network training.
The Annals of StatisticsApproximation of Density Functions by Sequences of Exponential Families
219 Citations1991Andrew R. Barron, Chyong-Hwa Sheu
Analogous convergence results for the relative entropy are shown to hold in general, for any class of log-density functions and sequence of finite-dimensional linear spaces having L2 and L.
IEEE Transactions on Pattern Analysis and Machine IntelligenceAutomatic pattern recognition: a study of the probability of error
144 Citations1988Luc Devroye
The Vapnik-Chervonenkis method can be used to choose the smoothing parameter in kernel-based rules, to choose k in the k-nearest neighbor rule, and to choose between parametric and nonparametric rules.
Statistical properties of artificial neural networks
80 Citations2003Andrew R. Barron
It is shown how, with the use of an appropriate complexity regularization criterion, the statistical risk of network estimators converges to zero as the sample size increases.
The Annals of StatisticsApproximation of Least Squares Regression on Nested Subspaces
55 Citations1988Dennis D. Cox
The Annals of Mathematical StatisticsOn the Tchebychef Inequality of Bernstein
49 Citations1933C. C. Craig
Neural Information Processing SystemsPractical Characteristics of Neural Network and Conventional Pattern Classifiers on Artificial and Speech Problems
45 Citations1989Yuchun Lee, Richard P. Lippmann
The results suggest that classifier selection should often depend more heavily on practical considerations concerning memory and computation resources, and restrictions on training and classification times than on error rate.