Convex Neural Networks
Published 5 December 2005
Yoshua Bengio, Nicolas Le Roux, Pascal Vincent, Olivier Delalleau, Patrice Marcotte
Citations138
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Training multi-layer neural networks in which the number of hidden units is learned can be viewed as a convex optimization problem, which involves an infinite number of variables but can be solved by incrementally inserting a hidden unit at a time.
Abstract
Convexity has recently received a lot of attention in the machine learning community, and the lack of convexity has been seen as a major disad-vantage of many learning algorithms, such as multi-layer artificial neural networks. We show that training multi-layer neural networks in which the number of hidden units is learned can be viewed as a convex optimization problem. This problem involves an infinite number of variables, but can be solved by incrementally inserting a hidden unit at a time, each time finding a linear classifier that minimizes a weighted sum of errors. 1
Keywords
Computer ScienceEngineering
Journal of the Royal Statistical Society Series B (Statistical Methodology)Regression Shrinkage and Selection Via the Lasso
51,790 Citations1996Robert Tibshirani
A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
NatureLearning representations by back-propagating errors
30,885 Citations1986David E. Rumelhart, Geoffrey E. Hinton +1 more
Back-propagation repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector, which helps to represent important features of the task domain.
The Annals of StatisticsGreedy function approximation: A gradient boosting machine.
28,973 Citations2001Jerome H. Friedman
A general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion, and specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification.
Journal of Computer and System SciencesA Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
20,320 Citations1997Yoav Freund, Robert E. Schapire
IEEE Transactions on Signal ProcessingMatching pursuits with time-frequency dictionaries
9,115 Citations1993Stéphane Mallat, Zhifeng Zhang
The authors introduce an algorithm, called matching pursuit, that decomposes any signal into a linear expansion of waveforms that are selected from a redundant dictionary of functions, chosen in order to best match the signal structures.
Journal of the American Statistical AssociationProjection Pursuit Regression
2,092 Citations1981Jerome H. Friedman, Werner Stuetzle
SIAM ReviewSemi-Infinite Programming: Theory, Methods, and Applications
1,005 Citations1993R. Hettich, K. O. Kortanek
This paper treats numerical methods based on either discretization or local reduction with the emphasis on the design of superlinearly convergent (SQP-type) methods.
Boosting Algorithms as Gradient Descent
712 Citations1999Llew Mason, Jonathan Baxter +2 more
Following previous theoretical results bounding the generalization performance of convex combinations of classifiers in terms of general cost functions of the margin, a new algorithm (DOOM II) is presented for performing a gradient descent optimization of such cost functions.
IEEE Transactions on Neural NetworksPerceptron-based learning algorithms
551 Citations1990Stephen I. Gallant
The heart of these algorithms is the pocket algorithm, a modification of perceptron learning that makes perceptronLearning well-behaved with nonseparable training data, even if the data are noisy and contradictory.
Journal of Physics A Mathematical and GeneralLearning in feedforward layered networks: the tiling algorithm
431 Citations1989Marc Mézard, Jean‐Pierre Nadal
A new algorithm which builds a feedforward layered network in order to learn any Boolean function of N Boolean units, which is an algorithm for growth of the network, which adds layers, and units inside a layer, at will until convergence.
Érudit documents and data repository (Érudit Consortium, University of Montreal)Efficient Non-Parametric Function Induction in Semi-Supervised Learning
173 Citations2004Yoshua Bengio, Olivier Delalleau +1 more
Experiments show that the proposed non-parametric algorithms which provide an estimated continuous label for the given unlabeled examples are extended to function induction algorithms that correspond to the minimization of a regularization criterion applied to an out-of-sample example, and happens to have the form of a Parzen windows regressor.
Perspectives in neural computingLeast Absolute Shrinkage is Equivalent to Quadratic Penalization
103 Citations1998Yves Grandvalet
Machine LearningSparse Regression Ensembles in Infinite and Finite Hypothesis Spaces
52 Citations2002Gunnar Rätsch, Ayhan Demiriz +1 more
There exists an optimal solution to the infinite hypothesis space problem consisting of a finite number of hypothesis, and two algorithms for solving the infinite and finite hypothesis problems are proposed.
IEEE Transactions on Neural NetworksOn convergence properties of pocket algorithm
29 Citations1997Marco Muselli
A modified version of the learning method considered, called pocket algorithm with ratchet, is shown to obtain an optimal configuration within a finite number of iterations independently of the given training set.
Mathematical Methods of Operations ResearchNovel approaches to the discrimination problem
21 Citations1992Patrice Marcotte, Gilles Savard
Two criteria for judging the quality of a candidate hyperplane are analyzed and the maximal distance of a misclassified point to the hyperplane and the number of misclassified points are analyzed.