MixMatch: A Holistic Approach to Semi-Supervised Learning
arXiv (Cornell University)Published 6 May 2019Open access
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel
Citations604
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work unify the current dominant approaches for semi-supervised learning to produce a new algorithm, MixMatch, that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeling data using MixUp.
Abstract
Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets. In this work, we unify the current dominant approaches for semi-supervised learning to produce a new algorithm, MixMatch, that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp. We show that MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts. For example, on CIFAR-10 with 250 labels, we reduce error rate by a factor of 4 (from 38% to 11%) and by a factor of 2 on STL-10. We also demonstrate how MixMatch can help achieve a dramatically better accuracy-privacy trade-off for differential privacy. Finally, we perform an ablation study to tease apart which components of MixMatch are most important for its success.
Keywords
Computer Science
Deep Learning with Differential Privacy
5,871 Citations2016Martı́n Abadi, Andy Chu +5 more
This work develops new algorithmic techniques for learning and a refined analysis of privacy costs within the framework of differential privacy, and demonstrates that deep neural networks can be trained with non-convex objectives, under a modest privacy budget, and at a manageable cost in software complexity, training efficiency, and model quality.
arXiv (Cornell University)mixup: Beyond Empirical Risk Minimization
4,767 Citations2017Hongyi Zhang, Moustapha Cissé +2 more
This work proposes mixup, a simple learning principle that trains a neural network on convex combinations of pairs of examples and their labels, which improves the generalization of state-of-the-art neural network architectures.
Reading digits in natural images with unsupervised feature learning
4,546 Citations2024Yuval Netzer
A new benchmark dataset for research use is introduced containing over 600,000 labeled digits cropped from Street View images, and variants of two recently proposed unsupervised feature learning methods are employed, finding that they are convincingly superior on benchmarks.
The MIT Press eBooksLaplacian Eigenmaps and Spectral Techniques for Embedding and Clustering
4,535 Citations2002Mikhail Belkin, Partha Niyogi
The algorithm provides a computationally efficient approach to nonlinear dimensionality reduction that has locality preserving properties and a natural connection to clustering.
The MIT Press eBooksSemi-Supervised Learning
4,308 Citations2006Olivier Chapelle, Schölkopf, B. +1 more
This first comprehensive overview of semi-supervised learning presents state-of-the-art algorithms, a taxonomy of the field, selected applications, benchmark experiments, and perspectives on ongoing and future research.
arXiv (Cornell University)Simple and Scalable Predictive Uncertainty Estimation using Deep\n Ensembles
3,004 Citations2016Balaji Lakshminarayanan, Alexander Pritzel +1 more
IEEE Transactions on Pattern Analysis and Machine IntelligenceVirtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
2,766 Citations2018Takeru Miyato, Shin‐ichi Maeda +2 more
A new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input that achieves state-of-the-art performance for semi-supervised learning tasks on SVHN and CIFAR-10.
Transductive Inference for Text Classification using Support Vector Machines
2,717 Citations1999Thorsten Joachims
An analysis of why TSVMs are well suited for text classi(cid:12)cation is presented, and an algorithm for training TSVMs e(cid:14)-ciently, handling 10,000 examples and more is proposed.
arXiv (Cornell University)Improved Regularization of Convolutional Neural Networks with Cutout
2,711 Citations2017Terrance DeVries, Graham W. Taylor
This paper shows that the simple regularization technique of randomly masking out square regions of input during training, which is called cutout, can be used to improve the robustness and overall performance of convolutional neural networks.
An analysis of single-layer networks in unsupervised feature learning
2,457 Citations2011Adam Coates, Andrew Y. Ng +1 more
arXiv (Cornell University)Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
1,591 Citations2017Antti Tarvainen, Harri Valpola
Fixing Weight Decay Regularization in Adam
1,137 Citations2018Ilya Loshchilov, Frank Hutter
This work decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets.
Neural ComputationDeep, Big, Simple Neural Nets for Handwritten Digit Recognition
1,064 Citations2010Dan Claudiu Cireşan, Ueli Meier +2 more
Good old online backpropagation for plain multilayer perceptrons yields a very low 0.35 error rate on the MNIST handwritten digits benchmark.
Semi-supervised Learning by Entropy Minimization.
1,021 Citations2005Yves Grandvalet, Yoshua Bengio
This framework, which motivates minimum entropy regularization, enables to incorporate unlabeled data in the standard supervised learning, and includes other approaches to the semi-supervised problem as particular or limiting cases.
Keeping the neural networks simple by minimizing the description length of the weights
957 Citations1993Geoffrey E. Hinton, Drew van Camp
A method of computing the derivatives of the expected squared error and of the amount of information in the noisy weights in a network that contains a layer of non-linear hidden units without time-consuming Monte Carlo simulations is described.
arXiv (Cornell University)AutoAugment: Learning Augmentation Policies from Data
731 Citations2018Ekin D. Cubuk, Barret Zoph +3 more
This paper describes a simple procedure called AutoAugment to automatically search for improved data augmentation policies, which achieves state-of-the-art accuracy on CIFAR-10, CIFar-100, SVHN, and ImageNet (without additional data).
Transductive learning via spectral graph partitioning
649 Citations2003Thorsten Joachims
This work proposes an algorithm that robustly achieves good generalization performance and that can be trained efficiently, and shows a connection to transductive Support Vector Machines, and that an effective Co-Training algorithm arises as a special case.
arXiv (Cornell University)Semi-Supervised Learning with Ladder Networks
596 Citations2015Antti Rasmus, Harri Valpola +3 more
This work builds on top of the Ladder network proposed by Valpola which is extended by combining the model with supervision and shows that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification in addition to permutation-invariant MNIST classification with all labels.
arXiv (Cornell University)Semi-Supervised Learning with Generative Adversarial Networks
541 Citations2016Augustus Odena
This work extends Generative Adversarial Networks to the semi-supervised context by forcing the discriminator network to output class labels and shows that this method can be used to create a more data-efficient classifier and that it allows for generating higher quality samples than a regular GAN.
The Importance of Encoding Versus Training with Sparse Coding and Vector Quantization
526 Citations2011Adam Coates, Andrew Y. Ng
This work investigates the reasons for the success of sparse coding over VQ by decoupling these phases, allowing us to separate out the contributions of training and encoding in a controlled way and shows not only that it can use fast VQ algorithms for training, but that they can just as well use randomly chosen exemplars from the training set.
arXiv (Cornell University)Decoupled Weight Decay Regularization
494 Citations2017Ilya Loshchilov, Frank Hutter
arXiv (Cornell University)Variational Autoencoder for Deep Learning of Images, Labels and Captions
372 Citations2016Yunchen Pu, Zhe Gan +5 more
Principled Hybrids of Generative and Discriminative Models
339 Citations2006Julia Lasserre, C.M. Bishop +1 more
A new perspective is adopted which says that there is only one correct way to train a given model, and that a ‘discriminatively trained’ generative model is fundamentally a new model and opens door to very general ways of interpolating between generative and discriminative extremes through alternative choices of prior.
Neural Information Processing SystemsRegularization with stochastic transformations and perturbations for deep semi-supervised learning
315 Citations2016Mehdi S. M. Sajjadi, Mehran Javanmardi +1 more
An unsupervised loss function is proposed that takes advantage of the stochastic nature of these methods and minimizes the difference between the predictions of multiple passes of a training sample through the network.
arXiv (Cornell University)Shake-Shake regularization
312 Citations2017Xavier Gastaldi
The method introduced in this paper aims at helping deep learning practitioners faced with an overfit problem to replace, in a multi-branch network, the standard summation of parallel branches with a stochastic affine combination.
arXiv (Cornell University)Scalable Private Learning with PATE
247 Citations2018Nicolas Papernot, Shuang Song +4 more
arXiv (Cornell University)Stacked What-Where Auto-encoders
228 Citations2015Junbo Zhao, Michaël Mathieu +2 more
A novel architecture, the "stacked what-where auto-encoders" (SWWAE), which integrates discriminative and generative pathways and provides a unified approach to supervised, semi-supervised and unsupervised learning without relying on sampling during training.
Neural Information Processing SystemsUsing Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes
188 Citations2007Geoffrey E. Hinton, Ruslan Salakhutdinov
This work shows how to use unlabeled data and a deep belief net (DBN) to learn a good covariance kernel for a Gaussian process.
Neural NetworksInterpolation Consistency Training for Semi-supervised Learning
137 Citations2019Vikas Verma, Alex Lamb +5 more
The theoretical analysis shows that Interpolation Consistency Training corresponds to a certain type of data-adaptive regularization with unlabeled points which reduces overfitting to labeled points under high confidence values.
arXiv (Cornell University)Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks
93 Citations2016Emily Denton, Sam Gross +1 more
A simple semi-supervised learning approach for images based on in-painting using an adversarial loss is introduced, able to directly train large VGG-style networks in a semi- supervised fashion.
Invariant Information Clustering for Unsupervised Image Classification and Segmentation
88 Citations2019Xu Ji, Andrea Vedaldi +1 more
A novel clustering objective is presented that learns a neural network classifier from scratch, given only unlabelled data samples, and discovers clusters that accurately match semantic classes, achieving state-of-the-art results in eight unsupervised clustering benchmarks spanning image classification and segmentation.
arXiv (Cornell University)Three Mechanisms of Weight Decay Regularization
55 Citations2018Guodong Zhang, Chaoqi Wang +2 more
arXiv (Cornell University)Invariant Information Distillation for Unsupervised Image Segmentation and Clustering
52 Citations2018Xu Ji, João F. Henriques +1 more
A simple loss based on information theory is used to extract meaningful representations directly from raw images, making it the first end-to-end unsupervised learning method that learns densely and efficiently (i.e. without sampling) for semantic segmentation.
arXiv (Cornell University)Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery
42 Citations2012Ian Goodfellow, Aaron Courville +1 more
This work derives a structured variational inference procedure and employs a variational EM training algorithm to improve upon the supervised learning capabilities of both sparse coding and the ssRBM on the CIFAR-10 dataset.
arXiv (Cornell University)Improving Consistency-Based Semi-Supervised Learning with Weight Averaging.
28 Citations2018Ben Athiwaratkun, Marc Finzi +2 more
arXiv (Cornell University)On the Generalization Properties of Differential Privacy
24 Citations2015Kobbi Nissim, Uri Stemmer
This work greatly simplifies the results of Dwork et al. and improves on the bound by showing that differential privacy guarantees O(\epsilon)$ accuracy with probability $1-O(\delta\log(1/\ep Silon)/\ep silon)$.
arXiv (Cornell University)Deep Metric Transfer for Label Propagation with Limited Annotated Data
12 Citations2018斌 刘, Zhirong Wu +2 more