NIPS 2016 Tutorial: Generative Adversarial Networks
arXiv (Cornell University)Published 31 December 2016Open access
Ian Goodfellow
Citations1,320
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This report summarizes the tutorial presented by the author at NIPS 2016 on generative adversarial networks (GANs), and describes state-of-the-art image models that combine GANs with other methods.
Abstract
This report summarizes the tutorial presented by the author at NIPS 2016 on generative adversarial networks (GANs). The tutorial describes: (1) Why generative modeling is a topic worth studying, (2) how generative models work, and how GANs compare to other generative models, (3) the details of how GANs work, (4) research frontiers in GANs, and (5) state-of-the-art image models that combine GANs with other methods. Finally, the tutorial contains three exercises for readers to complete, and the solutions to these exercises.
Keywords
Computer Science
UvA-DARE (University of Amsterdam)Adam: A Method for Stochastic Optimization
84,783 Citations2014Diederik P. Kingma, Jimmy Ba
2009 IEEE Conference on Computer Vision and Pattern RecognitionImageNet: A large-scale hierarchical image database
61,841 Citations2009Jia Deng, Wei Dong +4 more
A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
NatureMastering the game of Go with deep neural networks and tree search
15,831 Citations2016David Silver, Aja Huang +18 more
Using this search algorithm, the program AlphaGo achieved a 99.8% winning rate against other Go programs, and defeated the human European Go champion by 5 games to 0.5, the first time that a computer program has defeated a human professional player in the full-sized game of Go.
arXiv (Cornell University)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
15,635 Citations2024Sergey Ioffe
arXiv (Cornell University)Explaining and Harnessing Adversarial Examples
8,153 Citations2014Ian Goodfellow, Jonathon Shlens +1 more
arXiv (Cornell University)Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
6,981 Citations2015Alec Radford, Luke Metz +1 more
Cognitive ScienceA Learning Algorithm for Boltzmann Machines*
3,324 Citations1985David H. Ackley, Geoffrey E. Hinton +1 more
arXiv (Cornell University)Categorical Reparameterization with Gumbel-Softmax
3,217 Citations2016Eric Jang, Shixiang Gu +1 more
arXiv (Cornell University)Stochastic Backpropagation and Approximate Inference in Deep Generative Models
2,644 Citations2014Danilo Jimenez Rezende, Shakir Mohamed +1 more
Cambridge University Press eBooksGenerative Adversarial Networks
2,576 Citations2022Ian J. Goodfellow
arXiv (Cornell University)WaveNet: A Generative Model for Raw Audio
2,459 Citations2016Aäron van den Oord, Sander Dieleman +7 more
WaveNet, a deep neural network for generating raw audio waveforms, is introduced; it is shown that it can be efficiently trained on data with tens of thousands of samples per second of audio, and can be employed as a discriminative model, returning promising results for phoneme recognition.
Deep Boltzmann machines
1,778 Citations2009Ruslan Salakhutdinov, Geoffrey E. Hinton
A new learning algorithm for Boltzmann machines that contain many layers of hidden variables that is made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass.
Image-to-Image Translation with Conditional Adversarial Networks
1,561 Citations2017Phillip Isola, Jun-Yan Zhu +2 more
Conditional adversarial networks are investigated as a general-purpose solution to image-to-image translation problems and it is demonstrated that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks.
arXiv (Cornell University)Improved Techniques for Training GANs
1,386 Citations2016Tim Salimans, Ian Goodfellow +4 more
This work focuses on two applications of GANs: semi-supervised learning, and the generation of images that humans find visually realistic, and presents ImageNet samples with unprecedented resolution and shows that the methods enable the model to learn recognizable features of ImageNet classes.
Edinburgh Research Explorer (University of Edinburgh)Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
1,371 Citations2010Michael U. Gutmann, Aapo Hyvärinen
A new estimation principle is presented to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise, using the model log-density function in the regression nonlinearity, which leads to a consistent (convergent) estimator of the parameters.
arXiv (Cornell University)NICE: Non-linear Independent Components Estimation
1,340 Citations2014Laurent Dinh, David Krueger +1 more
Ghent University Academic Bibliography (Ghent University)InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
1,244 Citations2016Xi Chen, Yan Duan +4 more
Experiments show that InfoGAN learns interpretable representations that are competitive with representations learned by existing fully supervised methods.
MPG.PuRe (Max Planck Society)Generative Adversarial Text to Image Synthesis
1,083 Citations2016Scott Reed, Zeynep Akata +4 more
Graphical ModelsLearning and Relearning in Boltzmann Machines
1,052 Citations2001Geoffrey E. Hinton, T. J. Sejnowski
This chapter contains sections titled: Relaxation Searches, Easy and Hard learning, The Boltzmann Machine Learning Algorithm, An Example of Hard Learning, Achieving Reliable Computation with Unreliable Hardware, and an Example of the Effects of Damage.
arXiv (Cornell University)Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
1,041 Citations2016Christian Ledig, Lucas Theis +9 more
Trends in Cognitive SciencesLearning multiple layers of representation
990 Citations2007Geoffrey E. Hinton
The limitations of backpropagation learning can now be overcome by using multilayer neural networks that contain top-down connections and training them to generate sensory data rather than to classify it.
arXiv (Cornell University)Adversarial Feature Learning
723 Citations2016Jeff Donahue, Philipp Krähenbühl +1 more
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
704 Citations2017Han Zhang, Tao Xu +5 more
This paper proposes Stacked Generative Adversarial Networks (StackGAN) to generate 256 photo-realistic images conditioned on text descriptions and introduces a novel Conditioning Augmentation technique that encourages smoothness in the latent conditioning manifold.
arXiv (Cornell University)Adversarially Learned Inference
692 Citations2016Vincent Dumoulin, Ishmael Belghazi +5 more
arXiv (Cornell University)f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
635 Citations2016Sebastian Nowozin, Botond Cseke +1 more
Oxford University Research Archive (ORA) (University of Oxford)The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
618 Citations2016Chris J. Maddison, Andriy Mnih +1 more
Concrete random variables---continuous relaxations of discrete random variables is a new family of distributions with closed form densities and a simple reparameterization, and the effectiveness of Concrete relaxations on density estimation and structured prediction tasks using neural networks is demonstrated.
arXiv (Cornell University)Semi-Supervised Learning with Generative Adversarial Networks
541 Citations2016Augustus Odena
This work extends Generative Adversarial Networks to the semi-supervised context by forcing the discriminator network to output class labels and shows that this method can be used to create a more data-efficient classifier and that it allows for generating higher quality samples than a regular GAN.
Lecture notes in computer scienceWhat Does Classifying More Than 10,000 Image Categories Tell Us?
498 Citations2010Jia Deng, Alexander C. Berg +2 more
A study of large scale categorization including a series of challenging experiments on classification with more than 10,000 image classes finds that computational issues become crucial in algorithm design and conventional wisdom from a couple of hundred image categories does not necessarily hold when the number of categories increases.
Computers & Mathematics with ApplicationsGraphical models for machine learning and digital communication
465 Citations1999
arXiv (Cornell University)A note on the evaluation of generative models
435 Citations2015Lucas Theis, Aäron van den Oord +1 more
arXiv (Cornell University)Generative Adversarial Imitation Learning
368 Citations2016Jonathan Ho, Stefano Ermon
arXiv (Cornell University)Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
337 Citations2015Jost Tobias Springenberg
arXiv (Cornell University)Generative Moment Matching Networks
243 Citations2015Yujia Li, Kevin Swersky +1 more
This work forms a method that generates an independent sample via a single feedforward pass through a multilayer perceptron, as in the recently proposed generative adversarial networks, using MMD to learn to generate codes that can then be decoded to produce samples.
arXiv (Cornell University)Training generative neural networks via Maximum Mean Discrepancy\n optimization
222 Citations2015Gintare Karolina Dziugaite, Daniel M. Roy +1 more
This work considers training a deep neural network to generate samples from an unknown distribution given i.i.d. data to frame learning as an optimization minimizing a two-sample test statistic, and proves bounds on the generalization error incurred by optimizing the empirical MMD.
Zenodo (CERN European Organization for Nuclear Research)Neural Photo Editing With Introspective Adversarial Networks
219 Citations2017Andrew Brock, Theodore Lim +2 more
Characterization and computation of local Nash equilibria in continuous games
214 Citations2013Lillian J. Ratliff, Samuel A. Burden +1 more
Drawing on this analogy, an iterative steepest descent algorithm is proposed for numerical approximation of local Nash equilibria and a sufficient condition ensuring local convergence of the algorithm is provided.
arXiv (Cornell University)A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
214 Citations2016Chelsea Finn, Paul F. Christiano +2 more
It is shown that certain IRL methods are in fact mathematically equivalent to GANs, and an equivalence between a sample-based algorithm for maximum entropy IRL and a GAN in which the generator's density can be evaluated and is provided as an additional input to the discriminator.
Massively parallel architectures for AI: netl, thistle, and boltzmann machines
184 Citations1983Scott E. Fahlman, Geoffrey E. Hinton +1 more
Saccharides assessed as combined cryoprotectant, preservative and prebiotic of Lactobacillus plantarum and Inulin, with best combined performance using sucrose with storage at 4 °C.
arXiv (Cornell University)Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
157 Citations2016Anh‐Tu Nguyen, Jeff Clune +3 more
arXiv (Cornell University)Generative Visual Manipulation on the Natural Image Manifold
121 Citations2016Jun-Yan Zhu, Philipp Krähenbühl +2 more
arXiv (Cornell University)Learning to Protect Communications with Adversarial Neural Cryptography
119 Citations2016Martı́n Abadi, David G. Andersen
It is demonstrated that the neural networks can learn how to perform forms of encryption and decryption, and also how to apply these operations selectively in order to meet confidentiality goals.
arXiv (Cornell University)Deep Generative Stochastic Networks Trainable by Backprop
108 Citations2013Yoshua Bengio, Éric Thibodeau-Laufer +2 more
arXiv (Cornell University)Censoring Representations with an Adversary
104 Citations2015Harrison Edwards, Amos Storkey
This work forms the adversarial model as a minimax problem, and optimize that minimax objective using a stochastic gradient alternate min-max optimizer, and demonstrates the ability to provide discriminant free representations for standard test problems, and compares with previous state of the art methods for fairness.
arXiv (Cornell University)Connecting Generative Adversarial Networks and Actor-Critic Methods
93 Citations2016David Pfau, Oriol Vinyals
It is shown that GANs can be viewed as actor-critic methods in an environment where the actor cannot affect the reward, and it is hoped that by highlighting this formal connection it will encourage both GAN and RL communities to develop general, scalable, and stable algorithms for multilevel optimization with deep networks.
arXiv (Cornell University)On distinguishability criteria for estimating generative models
87 Citations2014Ian Goodfellow
arXiv (Cornell University)Unsupervised Learning of Visual Structure using Predictive Generative Networks
82 Citations2015William Lotter, Gabriel Kreiman +1 more
It is argued that prediction can serve as a powerful unsupervised loss for learning rich internal representations of high-level object features in deep neural networks trained using a CNN-LSTM-deCNN framework.
Generating Interpretable Images with Controllable Structure
56 Citations2017Scott Reed, Aäron van den Oord +4 more
Improved text-to-image synthesis with controllable object locations using an extension of Pixel Convolutional Neural Networks (PixelCNN) and it is shown how the model can generate images conditioned on part keypoints and segmentation masks.
arXiv (Cornell University)Fast Gradient-Based Inference with Continuous Latent Variable Models in Auxiliary Form
29 Citations2013Diederik P. Kingma
It is shown that, in many cases, it is possible to express Bayesian models in an auxiliary form, where continuous latent variables are conditionally deterministic given their parents and a set of independent auxiliary variables, leading to significant speedups in gradient-based inference.
neural information processing systemsHigher Order Statistical Decorrelation without Information Loss
26 Citations1994Gustavo Deco, Wilfried Brauer
A neural network learning paradigm based on information theory is proposed as a way to perform in an unsupervised fashion, redundancy reduction among the elements of the output layer without loss of information from the sensory input.
arXiv (Cornell University)Deep Visual Foresight for Planning Robot Motion
19 Citations2016Chelsea Finn, Sergey Levine