Pylearn2: a machine learning research library
arXiv (Cornell University)Published 20 August 2013Open access
Ian Goodfellow, David Warde-Farley, Pascal Lamblin, Vincent Dumoulin, Mehdi Mirza, Razvan Pascanu
Citations193
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A brief history of the library, an overview of its basic philosophy, a summary of the Library's architecture, and a description of how the Pylearn2 community functions socially are given.
Abstract
Pylearn2 is a machine learning research library. This does not just mean that it is a collection of machine learning algorithms that share a common API; it means that it has been designed for flexibility and extensibility in order to facilitate research projects that involve new or unusual use cases. In this paper we give a brief history of the library, an overview of its basic philosophy, a summary of the library's architecture, and a description of how the Pylearn2 community functions socially.
Keywords
Computer Science
Communications of the ACMImageNet classification with deep convolutional neural networks
75,709 Citations2017Alex Krizhevsky, Ilya Sutskever +1 more
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
arXiv (Cornell University)Scikit-learn: Machine Learning in Python
63,264 Citations2012Fabián Pedregosa, Gaël Varoquaux +17 more
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
Machine LearningSupport-Vector Networks
33,035 Citations1995Corinna Cortes, Vladimir Vapnik
High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated and the performance of the support- vector network is compared to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
Learning Internal Representations by Error Propagation
16,235 Citations1985David E. Rumelhart, Geoffrey E. Hinton +1 more
Annals of EugenicsTHE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS
14,727 Citations1936Ronald Aylmer Fisher
The London Edinburgh and Dublin Philosophical Magazine and Journal of ScienceLIII. <i>On lines and planes of closest fit to systems of points in space</i>
11,670 Citations1901Karl Pearson
This paper is concerned with the construction of planes of closest fit to systems of points in space and the relationships between these planes and the planes themselves.
Extracting and composing robust features with denoising autoencoders
7,343 Citations2008Pascal Vincent, Hugo Larochelle +2 more
This work introduces and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern.
arXiv (Cornell University)Improving neural networks by preventing co-adaptation of feature detectors
6,631 Citations2012Geoffrey E. Hinton, Nitish Srivastava +3 more
Deep Sparse Rectifier Neural Networks
5,408 Citations2012Xavier Glorot, Antoine Bordes +1 more
This paper shows that rectifying neurons are an even better model of biological neurons and yield equal or better performance than hyperbolic tangent networks in spite of the hard non-linearity and non-dierentiabil ity.
Neural ComputationTraining Products of Experts by Minimizing Contrastive Divergence
5,017 Citations2002Geoffrey E. Hinton
A product of experts (PoE) is an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary because it is hard even to approximate the derivatives of the renormalization term in the combination rule.
Reading digits in natural images with unsupervised feature learning
4,546 Citations2024Yuval Netzer
A new benchmark dataset for research use is introduced containing over 600,000 labeled digits cropped from Street View images, and variants of two recently proposed unsupervised feature learning methods are employed, finding that they are convincingly superior on benchmarks.
HAL (Le Centre pour la Communication Scientifique Directe)Convolutional networks for images, speech, and time series
4,359 Citations1998Yann LeCun, Yoshua Bengio
Computing in Science & EngineeringPython for Scientific Computing
3,293 Citations2007Travis E. Oliphant
Python is an excellent "steering" language for scientific codes written in other languages, but with additional basic tools, it transforms into a high-level language suited for scientific and engineering code that's often fastenough to be immediately useful but also flexible enough to be sped up with additional extensions.
Large Scale Distributed Deep Networks
2,916 Citations2012Jay B. Dean, Greg S. Corrado +10 more
This paper considers the problem of training a deep network with billions of parameters using tens of thousands of CPU cores and develops two algorithms for large-scale distributed training, Downpour SGD and Sandblaster L-BFGS, which increase the scale and speed of deep network training.
An analysis of single-layer networks in unsupervised feature learning
2,457 Citations2011Adam Coates, Andrew Y. Ng +1 more
Vision ResearchThe “independent components” of natural scenes are edge filters
2,269 Citations1997Anthony J. Bell, T. J. Sejnowski
It is shown that a new unsupervised learning algorithm based on information maximization, a nonlinear "infomax" network, when applied to an ensemble of natural scenes produces sets of visual filters that are localized and oriented.
SIAM Journal on Control and OptimizationAcceleration of Stochastic Approximation by Averaging
1,811 Citations1992B. T. Polyak, Anatoli Juditsky
Convergence with probability one is proved for a variety of classical optimization and identification problems and it is demonstrated for these problems that the proposed algorithm achieves the highest possible rate of convergence.
Deep Boltzmann machines
1,778 Citations2009Ruslan Salakhutdinov, Geoffrey E. Hinton
A new learning algorithm for Boltzmann machines that contain many layers of hidden variables that is made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass.
MIT Press eBooksInformation processing in dynamical systems: foundations of harmony theory
1,666 Citations1986Paul Smolensky
The work reported in this chapter rests on the conviction that a methodology that has a crucial role to play in the development of cognitive science is mathematical analysis.
Lecture notes in computer scienceChallenges in Representation Learning: A Report on Three Machine Learning Contests
1,483 Citations2013Ian Goodfellow, Dumitru Erhan +26 more
Edinburgh Research Explorer (University of Edinburgh)Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
1,371 Citations2010Michael U. Gutmann, Aapo Hyvärinen
A new estimation principle is presented to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise, using the model log-density function in the regression nonlinearity, which leads to a consistent (convergent) estimator of the parameters.
Biological CyberneticsAuto-association by multilayer perceptrons and singular value decomposition
1,357 Citations1988H. Bourlard, Y. Kamp
It is shown that, for auto-association, the nonlinearities of the hidden units are useless and that the optimal parameter values can be derived directly by purely linear techniques relying on singular value decomposition and low rank matrix approximation, similar in spirit to the well-known Karhunen-Loève transform.
Learning methods for generic object recognition with invariance to pose and lighting
1,332 Citations2004Yann LeCun, Fu Jie Huang +1 more
A real-time version of the system was implemented that can detect and classify objects in natural scenes at around 10 frames per second and proved impractical, while convolutional nets yielded 16/7% error.
Torch7: A Matlab-like Environment for Machine Learning
1,257 Citations2011Ronan Collobert, Koray Kavukcuoglu +1 more
Torch7 is a versatile numeric computing framework and machine learning library that extends Lua that can easily be interfaced to third-party software thanks to Lua’s light interface.
arXiv (Cornell University)Theano: new features and speed improvements
1,035 Citations2012Frédéric Bastien, Pascal Lamblin +7 more
New features and efficiency improvements to Theano are presented, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.
An empirical evaluation of deep architectures on problems with many factors of variation
1,030 Citations2007Hugo Larochelle, Dumitru Erhan +3 more
A series of experiments indicate that these models with deep architectures show promise in solving harder learning problems that exhibit many factors of variation.
Neural ComputationA Connection Between Score Matching and Denoising Autoencoders
980 Citations2011Pascal Vincent
A proper probabilistic model for the denoising autoencoder technique is defined, which makes it in principle possible to sample from them or rank examples by their energy, and a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives is suggested.
Training restricted Boltzmann machines using approximations to the likelihood gradient
857 Citations2008Tijmen Tieleman
A new algorithm for training Restricted Boltzmann Machines is introduced, which is compared to some standard Contrastive Divergence and Pseudo-Likelihood algorithms on the tasks of modeling and classifying various types of data.
Nonlinear Learning using Local Coordinate Coding
687 Citations2009Kai Yu, Tong Zhang +1 more
It is shown that a high dimensional nonlinear function can be approximated by a global linear function with respect to this coding scheme, and the approximation quality is ensured by the locality of such coding.
Maxout Networks
607 Citations2013Ian Goodfellow, David Warde-Farley +3 more
A simple new model called maxout is defined designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique.
Exponential Family Harmoniums with an Application to Information Retrieval
436 Citations2004Max Welling, Michal Rosen‐Zvi +1 more
An alternative two-layer model based on exponential family distributions and the semantics of undirected models is proposed, which performs well on document retrieval tasks and provides an elegant solution to searching with keywords.
Lecture notes in computer scienceHigher Order Contractive Auto-Encoder
375 Citations2011Salah Rifai, Grégoire Mesnil +5 more
A novel regularizer when training an autoencoder for unsupervised feature extraction yields representations that are significantly better suited for initializing deep architectures than previously proposed approaches, beating state-of-the-art performance on a number of datasets.
Lecture notes in computer scienceRank, Trace-Norm and Max-Norm
332 Citations2005Nathan Srebro, Adi Shraibman
This work studies the rank, trace-norm and max-norm as complexity measures of matrices, focusing on the problem of fitting a matrix with matrices having low complexity, and presents generalization error bounds for predicting unobserved entries that are based on these measures.
Stochastics and stochastics reportsOn the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity rates
142 Citations1999Laurent Younès
This work analyses the convergence of stochastic algorithms with Markovian noise when the ergodicity of the Markov chain governing the noise rapidly decreases as the control parameter tends to infinity and provides sufficient condition which ensure convergence.
International Conference on Artificial Intelligence and StatisticsA Spike and Slab Restricted Boltzmann Machine
77 Citations2011Aaron Courville, James Bergstra +1 more
The spike and slab Restricted Boltzmann Machine is introduced, characterized by having both a real-valued vector, the slab, and a binary variable, the spike, associated with each unit in the hidden layer.
International Conference on Machine LearningUnsupervised Models of Images by Spike-and-Slab RBMs
70 Citations2011Yoshua Bengio, Aaron Courville +1 more
This paper generalizes and extends the spike-and-slab RBM to include non-zero means of the conditional distribution over the observed variables given the binary spike variables, and introduces a term, quadratic in the observed data that is exploited to guarantee that all conditionals associated with the model are well defined.
Lecture notes in computer scienceInvariant Object Recognition with Slow Feature Analysis
57 Citations2008Mathias Franzius, Niko Wilbert +1 more
A model that is able to extract object identity, position, and rotation angles, where each code is independent of all others is proposed, which demonstrates the model behavior on complex three-dimensional objects under translation and in-depth rotation on homogeneous backgrounds.
IEEE Transactions on Pattern Analysis and Machine IntelligenceScaling Up Spike-and-Slab Models for Unsupervised Feature Learning
38 Citations2013Ian J. Goodfellow, Aaron Courville +1 more
It is demonstrated that using S3C as a feature extractor results in very good object recognition performance, particularly when the number of labeled examples is low, and that the PD-DBM may be trained successfully without greedy layerwise training.
International Conference on Machine LearningLarge-Scale Learning of Embeddings with Reconstruction Sampling
34 Citations2011Xavier Glorot, Yoshua Bengio +1 more
A novel method to speed up the learning of embeddings for large-scale learning tasks involving very sparse data, as is typically the case for Natural Language Processing tasks, using a new method to approximate reconstruction error by a sampling procedure.
Unsupervised and transfer learning challenge
33 Citations2011Isabelle Guyon, Gideon Dror +3 more
A data mining challenge in “unsupervised and transfer learning” (the UTL challenge) was organized, in collaboration with the DARPA Deep Learning program, to learn good data representations that can be re-used across tasks by building models that capture regularities of the input space.