A universal theorem on learning curves
Neural NetworksPublished 1 January 1993
Шун-ичи Амари
Citations97
SJR quartileQ1
SJR score1.49
SNIP2.02
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A universal asymptotic behavior of learning curves for general noiseless dichotomy machines, or neural networks is proved, it is proved that irrespective of the architecture of a machine, the average predictive entropy or the information gain converges to 0.
Abstract
A learning curve shows how fast a learning machine improves its behavior as the number of training examples increases. This paper proves a universal asymptotic behavior of learning curves for general noiseless dichotomy machines, or neural networks. It is proved that irrespective of the architecture of a machine, the average predictive entropy or the information gain 〈e∗(t)〉 converges to 0 as 〈e∗(t)〉 ∼ d/t as the number t of training exampies increases, where d is the number of modifiable parameters of a machine.
Keywords
Computer Science
A theory of the learnable
4,242 Citations1984Leslie G. Valiant
This paper regards learning as the phenomenon of knowledge acquisition in the absence of explicit programming, and gives a precise methodology for studying this phenomenon from a computational viewpoint.
Communications of the ACMA theory of the learnable
3,268 Citations1984Leslie G. Valiant
This paper regards learning as the phenomenon of knowledge acquisition in the absence of explicit programming, and gives a precise methodology for studying this phenomenon from a computational viewpoint.
Neural ComputationWhat Size Net Gives Valid Generalization?
1,555 Citations1989Eric B. Baum, David Haussler
It is shown that if m O(W/ ∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 2 ∊ of future test examples drawn from the same distribution.
IEEE Transactions on Neural NetworksNetwork information criterion-determining the number of hidden units for an artificial neural network model
649 Citations1994Noboru Murata, S. Yoshizawa +1 more
The problem of model selection, or determination of the number of hidden units, can be approached statistically, by generalizing Akaike's information criterion (AIC) to be applicable to unfaithful models with general loss criteria including regularization terms.
IEEE Transactions on Electronic ComputersA Theory of Adaptive Pattern Classifiers
520 Citations1967Шун-ичи Амари
It is proved that, by the procedures proposed here, the weight vector converges to the optimal one even under nonseparable pattern distributions, and there is an important tradeoff between speed and accuracy of convergence.
Proceedings of the IEEEA statistical approach to learning and generalization in layered neural networks
244 Citations1990Esther Levin, Naftali Tishby +1 more
The proposed formalism is applied to the problems of selecting an optimal architecture and the prediction of learning curves and the Gibbs distribution on the ensemble of networks with a fixed architecture is derived.
Physical Review LettersLearning from examples in large neural networks
157 Citations1990Haim Sompolinsky, Naftali Tishby +1 more
Numerical results on training in layered neural networks indicate that the generalization error improves gradually in some cases, and sharply in others, and statistical mechanics is used to study generalization curves in large layered networks.
Neural ComputationFour Types of Learning Curves
146 Citations1992Шун-ичи Амари, Naotake FUJITA +1 more
The present paper uses the Bayesian approach to show that given the annealed approximation, learning curves can be classified into four asymptotic types.
Neural ComputationStatistical Theory of Learning Curves under Entropic Loss Criterion
137 Citations1993Шун-ичи Амари, Noboru Murata
A universal property of learning curves is elucidated, which shows how the generalization error, training error, and the complexity of the underlying stochastic machine are related and how the behavior of a stochastics machine is improved as the number of training examples increases.
Physical Review ALearning processes in neural networks
105 Citations1991Tom Heskes, Bert Kappen
The ensemble description allows us to study the asymptotic behavior of the plasticities for a large class of neural networks and derives an expression for the size of the fluctuations in an unchanging environment for small learning parameters.
Predicting (0, 1)-functions on randomly drawn points
89 Citations1988David Haussler, N. Littlestone +1 more
The authors consider the problem of predicting (0, 1)-valued functions on R/sup n/ and smaller domains, based on their values on randomly drawn points, and construct prediction strategies that are optimal to within a constant factor for any reasonable class F of target functions.
Conference on Learning TheoryCalculation of the learning curve of Bayes optimal classification algorithm for learning a perceptron with noise
42 Citations1991Manfred Opper, David Haussler
Conference on Learning TheoryA loss bound model for on-line stochastic prediction strategies
12 Citations1991Kenji Yamanishi