Semi-Supervised Learning with Ladder Networks
Published 9 July 2015
Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, Tapani Raiko
Citations659
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation, avoiding the need for layer-wise pre-training. Our work builds on the Ladder network proposed by Valpola (2015), which we extend by combining the model with supervision. We show that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification, in addition to permutation-invariant MNIST classification with all labels.
Keywords
Computer Science
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)
50,318 Citations2021Mandi, Jayanta, Canoy, Rocsildes +2 more
A simple numeric simulation of DNA-co-polymerized hydrogel shape change and a genetic algorithm that generates and selects large batches of material designs that compete with one another to evolve and converge on optimal objective-matching designs are constructed.
Dropout: a simple way to prevent neural networks from overfitting
34,279 Citations2014Nitish Srivastava, Geoffrey E. Hinton +3 more
It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
arXiv (Cornell University)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
15,635 Citations2024Sergey Ioffe
Combining labeled and unlabeled data with co-training
5,604 Citations1998Avrim Blum, Tom M. Mitchell
Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
5,014 Citations2010Pascal Vincent, Hugo Larochelle +3 more
The MIT Press eBooksSemi-Supervised Learning
4,308 Citations2006Olivier Chapelle, Schölkopf, B. +1 more
This first comprehensive overview of semi-supervised learning presents state-of-the-art algorithms, a taxonomy of the field, selected applications, benchmark experiments, and perspectives on ongoing and future research.
Journal of the American Statistical AssociationStatistical Analysis of Finite Mixture Distributions.
2,940 Citations1987Bruce G. Lindsay, D. M. Titterington +2 more
arXiv (Cornell University)Striving for Simplicity: The All Convolutional Net
2,603 Citations2014Jost Tobias Springenberg, Alexey Dosovitskiy +2 more
Adaptive deconvolutional networks for mid and high level feature learning
1,249 Citations2011Matthew D. Zeiler, Graham W. Taylor +1 more
A hierarchical model that learns image decompositions via alternating layers of convolutional sparse coding and max pooling, relying on a novel inference scheme that ensures each layer reconstructs the input, rather than just the output of the layer directly beneath, as is common with existing hierarchical approaches.
arXiv (Cornell University)Discriminative Unsupervised Feature Learning with Convolutional Neural Networks
649 Citations2014Alexey Dosovitskiy, Jost Tobias Springenberg +2 more
This paper presents an approach for training a convolutional neural network using only unlabeled data and trains the network to discriminate between a set of surrogate classes, finding that this simple feature learning algorithm is surprisingly successful when applied to visual object recognition.
Lecture notes in computer scienceDeep Learning via Semi-supervised Embedding
568 Citations2012Jason Weston, Frédéric Ratle +2 more
Maxout Networks
451 Citations2013Ian Goodfellow, David Warde-Farley +3 more
Lecture notes in computer scienceHigher Order Contractive Auto-Encoder
375 Citations2011Salah Rifai, Grégoire Mesnil +5 more
A novel regularizer when training an autoencoder for unsupervised feature extraction yields representations that are significantly better suited for initializing deep architectures than previously proposed approaches, beating state-of-the-art performance on a number of datasets.
arXiv (Cornell University)Stacked What-Where Auto-encoders
228 Citations2015Junbo Zhao, Michaël Mathieu +2 more
A novel architecture, the "stacked what-where auto-encoders" (SWWAE), which integrates discriminative and generative pathways and provides a unified approach to supervised, semi-supervised and unsupervised learning without relying on sampling during training.
Semi-supervised learning of compact document representations with deep networks
223 Citations2008Marc ' Aurelio Ranzato, Martin Szummer
An algorithm to learn text document representations based on semi-supervised autoencoders that are stacked to form a deep network that can be trained efficiently on partially labeled corpora, producing very compact representations of documents, while retaining as much class information and joint word statistics as possible.
Elsevier eBooksFrom neural PCA to deep unsupervised learning
182 Citations2015Harri Valpola
A network supporting deep unsupervised learning is presented that is an autoencoder with lateral shortcut connections from the encoder to the decoder at each level of the hierarchy, analogous to hierarchical latent variable models.
CogPrints (Cogprints)Denoising Source Separation
176 Citations2005Jaakko Särelä, Harri Valpola
In the experimental section, various DSS schemes are applied extensively to artificial data, to real magnetoencephalograms and to simulated CDMA mobile network signals, and DSS appears to have relevance to many existing models of neural information processing.
arXiv (Cornell University)Distributional Smoothing with Virtual Adversarial Training
168 Citations2015Takeru Miyato, Shin‐ichi Maeda +3 more
Journal of the American Statistical AssociationIterative Reclassification Procedure for Constructing an Asymptotically Optimal Rule of Allocation in Discriminant Analysis
163 Citations1975Geoffrey J. McLachlan
arXiv (Cornell University)Blocks and Fuel: Frameworks for deep learning
125 Citations2015Bart van Merriënboer, Dzmitry Bahdanau +5 more
This work introduces two Python frameworks to train neural networks on large datasets: Blocks and Fuel, which provides a standard format for machine learning datasets.
arXiv (Cornell University)How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
119 Citations2014Yoshua Bengio
The motivations and theoretical justifications for this approach are laid down, along with conjectures that will have to be verified either mathematically or experimentally, including a hypothesis stating that such auto-encoder mediated target propagation could play in brains the role of credit assignment through many non-linear, noisy and discrete transformations.
Neural Information Processing SystemsMulti-Prediction Deep Boltzmann Machines
116 Citations2013Ian Goodfellow, Mehdi Mirza +2 more
The multi-prediction deep Boltzmann machine does not require greedy layerwise pretraining, and outperforms the standard DBM at classification, classification with missing inputs, and mean field prediction tasks.
arXiv (Cornell University)Deep AutoRegressive Networks
110 Citations2013Karol Gregor, Ivo Danihelka +3 more
An efficient approximate parameter estimation method based on the minimum description length (MDL) principle is derived, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference.
arXiv (Cornell University)Techniques for Learning Binary Stochastic Feedforward Neural Networks
87 Citations2015Tapani Raiko, Mathias Berglund +2 more
Lecture notes in computer scienceRule-injection hints as a means of improving network performance and learning time
71 Citations1990Steven C. Suddarth, Yannick Kergosien
The potential of the hint, whether applied to back-propagation learning or to more general types of pattern associators is to reduce training time and improve generalization performance.
Semi-supervised learning using an unsupervised atlas
23 Citations2014Nikolaos Pitelis, Charlotte Russell +1 more
arXiv (Cornell University)Denoising autoencoder with modulated lateral connections learns invariant representations of natural images
14 Citations2014Antti Rasmus, Tapani Raiko +1 more
Adding modulated lateral connections to the model improves the accuracy of the probability model for inputs, as measured by denoising performance, results in representations whose degree of invariance grows faster towards the higher layers, and supports the formation of diverse invariant poolings.
arXiv (Cornell University)Lateral Connections in Denoising Autoencoders Support Supervised Learning
11 Citations2015Antti Rasmus, Harri Valpola +1 more
The proposed model is trained to minimize simultaneously the sum of supervised and unsupervised cost functions by back-propagation, avoiding the need for layer-wise pretraining in the permutation-invariant MNIST classification task.