Learning with Pseudo-Ensembles
arXiv (Cornell University)Published 16 December 2014Open access
Phil Bachman, Ouais Alsharif, Doina Precup
Citations361
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it is presented, which naturally extends to the semi-supervised setting, where it produces state-of-the-art results.
Abstract
We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E.g., dropout (Hinton et. al, 2012) in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in the parent network. We present a novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it. In the fully-supervised setting, our regularizer matches the performance of dropout. But, unlike dropout, our regularizer naturally extends to the semi-supervised setting, where it produces state-of-the-art results. We provide a case study in which we transform the Recursive Neural Tensor Network of (Socher et. al, 2013) into a pseudo-ensemble, which significantly improves its performance on a real-world sentiment analysis benchmark.
Keywords
Computer ScienceMathematics
Extracting and composing robust features with denoising autoencoders
7,343 Citations2008Pascal Vincent, Hugo Larochelle +2 more
This work introduces and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern.
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
6,774 Citations2013Richard Socher, Alex Perelygin +5 more
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
arXiv (Cornell University)Improving neural networks by preventing co-adaptation of feature detectors
6,631 Citations2012Geoffrey E. Hinton, Nitish Srivastava +3 more
arXiv (Cornell University)Distributed Representations of Sentences and Documents
5,128 Citations2014Quoc V. Le, Tomáš Mikolov
Paragraph Vector is an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents, and its construction gives the algorithm the potential to overcome the weaknesses of bag-of-words models.
arXiv (Cornell University)On the difficulty of training Recurrent Neural Networks
3,778 Citations2012Razvan Pascanu, Tomáš Mikolov +1 more
This paper proposes a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem and validates empirically the hypothesis and proposed solutions.
A Convolutional Neural Network for Modelling Sentences
3,561 Citations2014Nal Kalchbrenner, Edward Grefenstette +1 more
A convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) is described that is adopted for the semantic modelling of sentences and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations.
DSpace@MIT (Massachusetts Institute of Technology)Theory and Applications of Robust Optimization
2,483 Citations2010David B. Brown, Constantine Caramanis +1 more
Lectures on Stochastic Programming: Modeling and Theory
1,732 Citations2009Alexander Shapiro, Darinka Dentcheva +1 more
Society for Industrial and Applied Mathematics eBooksLectures on Stochastic Programming: Modeling and Theory, Second Edition
553 Citations2014Alexander Shapiro, Darinka Dentcheva +1 more
The authors introduce new material to reflect recent developments in stochastic programming, including an analytical description of the tangent and normal cones of chance constrained sets and in-depth analysis of dynamic risk measures and concepts of time consistency.
Neural Information Processing SystemsUnderstanding Dropout
431 Citations2013Pierre Baldi, Peter Sadowski
A general formalism for studying dropout on either units or connections, with arbitrary probability values, is introduced and used to analyze the averaging and regularizing properties of dropout in both linear and non-linear networks.
Deep learning via semi-supervised embedding
387 Citations2008Jason Weston, Frédéric Ratle +1 more
It is shown how nonlinear embedding algorithms popular for use with shallow semi-supervised learning techniques such as kernel methods can be applied to deep multilayer architectures, either as a regularizer at the output layer, or on each layer of the architecture.
arXiv (Cornell University)Robustness and Regularization of Support Vector Machines
309 Citations2008Huan Xu, Constantine Caramanis +1 more
arXiv (Cornell University)Dropout Training as Adaptive Regularization
274 Citations2013Stefan Wager, Sida Wang +1 more
arXiv (Cornell University)Deep Generative Stochastic Networks Trainable by Backprop
238 Citations2013Yoshua Bengio, Eric Laufer +2 more
Theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders are provided and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood.
The Manifold Tangent Classifier
157 Citations2011Salah Rifai, Yann Dauphin +3 more
A representation learning algorithm can be stacked to yield a deep architecture and it is shown how it builds a topological atlas of charts, each chart being characterized by the principal singular vectors of the Jacobian of a representation mapping.
Learning with Marginalized Corrupted Features
143 Citations2013Laurens van der Maaten, Minmin Chen +2 more
This work proposes to corrupt training examples with noise from known distributions within the exponential family and presents a novel learning algorithm, called marginalized corrupted features (MCF), that trains robust predictors by minimizing the expected value of the loss function under the corrupting distribution.
arXiv (Cornell University)Large-Scale Feature Learning With Spike-and-Slab Sparse Coding
49 Citations2012Ian Goodfellow, Aaron Courville +1 more
This work introduces a new feature learning and extraction procedure based on a factor model the authors call spike-and-slab sparse coding (S3C), and presents a novel inference procedure for appropriate for use with GPUs which allows to dramatically increase both the training set size and the amount of latent factors that S3C may be trained with.
IEEE Transactions on Information TheoryRobust Regression and Lasso
48 Citations2010Huan Xu, Constantine Caramanis +1 more
It is shown that robustness of the solution explains why the solution is sparse, and a theorem is proved which states that sparsity and algorithmic stability contradict each other, and hence Lasso is not stable.
arXiv (Cornell University)Learning Ordered Representations with Nested Dropout
44 Citations2014Oren Rippel, Michael A. Gelbart +1 more
Nested dropout, a procedure for stochastically removing coherent nested sets of hidden units in a neural network, is introduced and it is rigorously shown that the application of nested dropout enforces identifiability of the units, which leads to an exact equivalence with PCA.