Four Types of Learning Curves
Neural ComputationPublished 1 July 1992
Шун-ичи Амари, Naotake FUJITA, Shigeru Shinomoto
Citations146
SJR quartileQ1
SJR score0.83
SNIP1.45
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The present paper uses the Bayesian approach to show that given the annealed approximation, learning curves can be classified into four asymptotic types.
Abstract
If machines are learning to make decisions given a number of examples, the generalization error ε(t) is defined as the average probability that an incorrect decision is made for a new example by a machine when trained with t examples. The generalization error decreases as t increases, and the curve ε(t) is called a learning curve. The present paper uses the Bayesian approach to show that given the annealed approximation, learning curves can be classified into four asymptotic types. If the machine is deterministic with noiseless teacher signals, then (1) ε ∼ at -1 when the correct machine parameter is unique, and (2) ε ∼ at -2 when the set of the correct parameters has a finite measure. If the teacher signals are noisy, then (3) ε ∼ at -1/2 for a deterministic machine, and (4) ε ∼ c + at -1 for a stochastic machine.
Keywords
Computer Science
Learning Internal Representations by Error Propagation
16,235 Citations1985David E. Rumelhart, Geoffrey E. Hinton +1 more
A theory of the learnable
4,242 Citations1984Leslie G. Valiant
This paper regards learning as the phenomenon of knowledge acquisition in the absence of explicit programming, and gives a precise methodology for studying this phenomenon from a computational viewpoint.
Communications of the ACMA theory of the learnable
3,268 Citations1984Leslie G. Valiant
This paper regards learning as the phenomenon of knowledge acquisition in the absence of explicit programming, and gives a precise methodology for studying this phenomenon from a computational viewpoint.
Lecture notes in statisticsDifferential-Geometrical Methods in Statistics
1,647 Citations1985Шун-ичи Амари
Neural ComputationWhat Size Net Gives Valid Generalization?
1,555 Citations1989Eric B. Baum, David Haussler
It is shown that if m O(W/ ∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 2 ∊ of future test examples drawn from the same distribution.
Neural ComputationLearning in Artificial Neural Networks: A Statistical Perspective
922 Citations1989Halbert White
Concepts and analytical results from the literatures of mathematical statistics, econometrics, systems identification, and optimization theory relevant to the analysis of learning in artificial neural networks are reviewed.
IEEE Transactions on Electronic ComputersA Theory of Adaptive Pattern Classifiers
520 Citations1967Шун-ичи Амари
It is proved that, by the procedures proposed here, the weight vector converges to the optimal one even under nonseparable pattern distributions, and there is an important tradeoff between speed and accuracy of convergence.
Physical Review AStatistical mechanics of learning from examples
513 Citations1992H. Sebastian Seung, Haim Sompolinsky +1 more
It is shown that for smooth networks, i.e., those with continuously varying weights and smooth transfer functions, the generalization curve asymptotically obeys an inverse power law, while for nonsmooth networks other behaviors can appear, depending on the nature of the nonlinearities as well as the realizability of the rule.
Proceedings of the IEEEA statistical approach to learning and generalization in layered neural networks
244 Citations1990Esther Levin, Naftali Tishby +1 more
The proposed formalism is applied to the problems of selecting an optimal architecture and the prediction of learning curves and the Gibbs distribution on the ensemble of networks with a fixed architecture is derived.
Proceedings of the IEEEMathematical foundations of neurocomputing
229 Citations1990Шун-ичи Амари
An advanced theory of learning and self-organization is proposed, covering backpropagation and its generalizations as well as the formation of topological maps and neural representations of information.
IEEE Transactions on Neural NetworksInformation geometry of Boltzmann machines
196 Citations1992Шун-ичи Амари, K. Kurata +1 more
Using the new theory of information geometry, a natural invariant Riemannian metric and a dual pair of affine connections on the Boltzmann neural network manifold are established and the meaning of geometrical structures is elucidated from the stochastic and the statistical point of view.
Physical Review LettersLearning from examples in large neural networks
157 Citations1990Haim Sompolinsky, Naftali Tishby +1 more
Numerical results on training in layered neural networks indicate that the generalization error improves gradually in some cases, and sharply in others, and statistical mechanics is used to study generalization curves in large layered networks.
Neural ComputationStatistical Theory of Learning Curves under Entropic Loss Criterion
137 Citations1993Шун-ичи Амари, Noboru Murata
A universal property of learning curves is elucidated, which shows how the generalization error, training error, and the complexity of the underlying stochastic machine are related and how the behavior of a stochastics machine is improved as the number of training examples increases.
Neural NetworksA universal theorem on learning curves
97 Citations1993Шун-ичи Амари
A universal asymptotic behavior of learning curves for general noiseless dichotomy machines, or neural networks is proved, it is proved that irrespective of the architecture of a machine, the average predictive entropy or the information gain converges to 0.
Neural ComputationExhaustive Learning
73 Citations1990Daniel Schwartz, Vijay K. Samalam +2 more
A simple Boolean learning problem involving networks with binary weights is numerically solved to obtain the entropy Sm and the average generalization ability Gm as a function of the size m of the training set.
Neural NetworksDualistic geometry of the manifold of higher-order neurons
70 Citations1991Шун-ичи Амари
An information geometrical method, which can be applied to more general neural network manifolds, is proposed and the accuracy of statistical estimation is shown in terms of the dimensionality of a model and the number of examples.
Neural ComputationThe Perceptron Algorithm is Fast for Nonmalicious Distributions
66 Citations1990Eric B. Baum
A modification of Valiant's distribution-independent protocol for learning is proposed in which the distribution and the function to be learned may be chosen by adversaries, however these adversaries may not communicate.
Europhysics Letters (EPL)Learning from Examples in a Single-Layer Neural Network
65 Citations1990David Hansel, Haim Sompolinsky
The average error, ε, made by the system in classifying novel inputs, as a function of the number of learnt examples is calculated and the importance of introducing errors in the learning of the examples is demonstrated.
Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topicsGeneralization in a two-layer neural network
32 Citations1993Kukjin Kang, Jong–Hoon Oh +2 more
Generalization in a fully connected two-layer neural network with N input nodes, M hidden nodes, a single output node, and binary weights is studied in the annealed approximation.
Neural ComputationLearning Curves for Error Minimum and Maximum Likelihood Algorithms
11 Citations1992Yoshiyuki Kabashima, Shigeru Shinomoto
This work addresses the question of how the asymptotic form of (t) as well as its limit of convergence reflect the choice of learning algorithms.