Deep learning via semi-supervised embedding
Published 1 January 2008
Jason Weston, Frédéric Ratle, Ronan Collobert
Citations387
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is shown how nonlinear embedding algorithms popular for use with shallow semi-supervised learning techniques such as kernel methods can be applied to deep multilayer architectures, either as a regularizer at the output layer, or on each layer of the architecture.
Abstract
We show how nonlinear embedding algorithms popular for use with shallow semi-supervised learning techniques such as kernel methods can be applied to deep multilayer architectures, either as a regularizer at the output layer, or on each layer of the architecture. This provides a simple alternative to existing approaches to deep learning whilst yielding competitive error rates compared to those methods, and existing shallow semi-supervised techniques.
Keywords
Computer Science
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
TechnometricsStatistical Learning Theory
26,913 Citations1999Yuhai Wu, Vladimir Vapnik
Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
ScienceA Global Geometric Framework for Nonlinear Dimensionality Reduction
13,740 Citations2000Joshua B. Tenenbaum, Vin de Silva +1 more
An approach to solving dimensionality reduction problems that uses easily measured local metric information to learn the underlying global geometry of a data set and efficiently computes a globally optimal solution, and is guaranteed to converge asymptotically to the true structure.
Neural ComputationLaplacian Eigenmaps for Dimensionality Reduction and Data Representation
7,709 Citations2003Mikhail Belkin, Partha Niyogi
This work proposes a geometrically motivated algorithm for representing the high-dimensional data that provides a computationally efficient approach to nonlinear dimensionality reduction that has locality-preserving properties and a natural connection to clustering.
PsychometrikaMultidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric Hypothesis
7,398 Citations1964Joseph B. Kruskal
The fundamental hypothesis is that dissimilarities and distances are monotonically related, and a quantitative, intuitively satisfying measure of goodness of fit is defined to this hypothesis.
Dimensionality Reduction by Learning an Invariant Mapping
5,057 Citations2006Raia Hadsell, Sumit Chopra +1 more
This work presents a method - called Dimensionality Reduction by Learning an Invariant Mapping (DrLIM) - for learning a globally coherent nonlinear function that maps the data evenly to the output manifold.
The MIT Press eBooksGreedy Layer-Wise Training of Deep Networks
4,704 Citations2007Yoshua Bengio, Pascal Lamblin +2 more
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.
The MIT Press eBooksSemi-Supervised Learning
4,308 Citations2006Olivier Chapelle, Schölkopf, B. +1 more
This first comprehensive overview of semi-supervised learning presents state-of-the-art algorithms, a taxonomy of the field, selected applications, benchmark experiments, and perspectives on ongoing and future research.
Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples
3,266 Citations2006Mikhail Belkin, Partha Niyogi +1 more
A semi-supervised framework that incorporates labeled and unlabeled data in a general-purpose learner is proposed and properties of reproducing kernel Hilbert spaces are used to prove new Representer theorems that provide theoretical basis for the algorithms.
International Journal of Pattern Recognition and Artificial IntelligenceSIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORK
2,096 Citations1993Jane Bromley, JAMES W. BENTZ +6 more
An algorithm for verification of signatures written on a pen-input tablet based on a novel, artificial neural network called a "Siamese" neural network, which consists of two identical sub-networks joined at their outputs.
Learning from Labeled and Unlabeled Data with Label Propagation
1,568 Citations2002Xiongjie Zhu, Zoubin Ghahramani
A simple iterative algorithm, label propagation, to propagate labels through the dataset along high density areas defined by unlabeled data is proposed and its solution is analyzed, and its connection to several other algorithms is analyzed.
Journal of Machine Learning ResearchA Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data
1,372 Citations2005Rie Kubota Ando, Tong Zhang
This paper presents a general framework in which the structural learning problem can be formulated and analyzed theoretically, and relate it to learning with unlabeled data, and algorithms for structural learning will be proposed, and computational issues will be investigated.
Multitask learning
1,118 Citations1998Rich Caruana
Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
1,104 Citations2007Marc’Aurelio Ranzato, Fu Jie Huang +2 more
An unsupervised method for learning a hierarchy of sparse feature detectors that are invariant to small shifts and distortions that alleviates the over-parameterization problems that plague purely supervised learning procedures, and yields good performance with very few labeled training samples.
Semi-Supervised Classification by Low Density Separation
710 Citations2005Olivier Chapelle, Alexander Zien +1 more
Three semi-supervised algorithms are proposed: deriving graph-based distances that emphazise low density regions between clusters, followed by training a standard SVM, and optimizing the Transductive SVM objective function by gradient descent.
Large Scale Transductive SVMs
463 Citations2006Ronan Collobert, Fabian H. Sinz +2 more
It is shown how the concave-convex procedure can be applied to transductive SVMs, which traditionally require solving a combinatorial search problem, and provides for the first time a highly scalable algorithm in the nonlinear case.
Cluster Kernels for Semi-Supervised Learning
461 Citations2002Olivier Chapelle, Jason Weston +1 more
A framework to incorporate unlabeled data in kernel classifier, based on the idea that two points in the same cluster are more likely to have the same label is proposed by modifying the eigenspectrum of the kernel matrix.
Beyond the point cloud
443 Citations2005Vikas Sindhwani, Partha Niyogi +1 more
This paper constructs a family of data-dependent norms on Reproducing Kernel Hilbert Spaces (RKHS) that allow the structure of the RKHS to reflect the underlying geometry of the data.
Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure
438 Citations2007Ruslan Salakhutdinov, Geoffrey E. Hinton
Shallow Semantic Parsing using Support Vector Machines.
368 Citations2004Sameer Pradhan, Wayne Ward +3 more
A machine learning al- algorithm for shallow semantic parsing based on Support Vector Machines which shows performance improvements through a number of new features and their ability to generalize to a new test set drawn from the AQUAINT corpus is measured.
Deep learning from temporal coherence in video
355 Citations2009Hossein Mobahi, Ronan Collobert +1 more
A learning method for deep architectures that takes advantage of sequential data, in particular from the temporal coherence that naturally exists in unlabeled video recordings, and is used to improve the performance on a supervised task of interest.
Machine LearningOn a Connection between Kernel PCA and Metric Multidimensional Scaling
205 Citations2002Christopher K. I. Williams
The kernel PCA algorithm of Schölkopf, Smola, and Müller can be interpreted as a form of metric multidimensional scaling (MDS) when the kernel function k(x, y) is isotropic, i.e. it depends only on ‖x − y‖.
Neural ComputationNonlinear Autoassociation Is Not Equivalent to PCA
170 Citations2000Nathalie Japkowicz, Stephen José Hanson +1 more
It is demonstrated that nonlinear autoassociators actually behave differently from linear methods and that they can outperform these methods when used for latent extraction, projection, and classification.
Neural ComputationLearning Optimized Features for Hierarchical Models of Invariant Object Recognition
168 Citations2003Heiko Wersing, Edgar Körner
This work proposes a feedforward model for recognition that shares components like weight sharing, pooling stages, and competitive nonlinearities with earlier approaches but focuses on new methods for learning optimal feature-detecting cells in intermediate stages of the hierarchical network.
Lecture notes in computer scienceTraining Hierarchical Feed-Forward Visual Recognition Models Using Transfer Learning from Pseudo-Tasks
151 Citations2008Amr Ahmed, Kai Yu +3 more
This paper presents a framework for training hierarchical feed-forward models for visual recognition, using transfer learning from pseudo tasks, and shows that these pseudo tasks induce an informative inverse-Wishart prior on the functional behavior of the network, offering an effective way to incorporate useful prior knowledge into the network training.
View-based 3D object recognition with support vector machines
100 Citations2003Danny Roobaert, Marc M. Van Hulle
This work reports high correct classification of unseen views, especially considering that no domain knowledge is including into the proposed system, and suggests an active learning algorithm to reduce further the required number of training views.
Fast Semantic Extraction Using a Novel Neural Network Architecture
63 Citations2007Ronan Collobert, Jason Weston
Large scale manifold transduction
45 Citations2008Michael Karlen, Jason Weston +2 more
This work shows how the regularizer of Transductive Support Vector Machines can be trained by stochastic gradient descent for linear models and multi-layer architectures, and proposes a natural generalization of the TSVM loss function that takes into account neighborhood and manifold information directly.