Image-to-Image Translation with Conditional Adversarial Networks
Published 1 July 2017Open access
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
Citations1,561
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Conditional adversarial networks are investigated as a general-purpose solution to image-to-image translation problems and it is demonstrated that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks.
Abstract
We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.
Keywords
Computer Science
Lecture notes in computer scienceU-Net: Convolutional Networks for Biomedical Image Segmentation
88,939 Citations2015Olaf Ronneberger, Philipp Fischer +1 more
It is shown that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks.
IEEE Transactions on Image ProcessingImage quality assessment: from error visibility to structural similarity
56,043 Citations2004Zhou Wang, Alan C. Bovik +2 more
A structural similarity index is developed and its promise is demonstrated through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)
50,318 Citations2021Mandi, Jayanta, Canoy, Rocsildes +2 more
A simple numeric simulation of DNA-co-polymerized hydrogel shape change and a genetic algorithm that generates and selects large batches of material designs that compete with one another to evolve and converge on optimal objective-matching designs are constructed.
International Journal of Computer VisionImageNet Large Scale Visual Recognition Challenge
40,207 Citations2015Olga Russakovsky, Jia Deng +10 more
The creation of this benchmark dataset and the advances in object recognition that have been possible as a result are described, and the state-of-the-art computer vision accuracy with human accuracy is compared.
Fully convolutional networks for semantic segmentation
36,888 Citations2015Jonathan Long, Evan Shelhamer +1 more
The key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning.
Journal of Japan Society for Fuzzy Theory and Intelligent InformaticsGAN（Generative Adversarial Nets）
21,715 Citations2017Ian Goodfellow
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
arXiv (Cornell University)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
15,635 Citations2024Sergey Ioffe
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
12,327 Citations2017Christian Ledig, Lucas Theis +9 more
SRGAN, a generative adversarial network (GAN) for image super-resolution (SR), is presented, to its knowledge, the first framework capable of inferring photo-realistic natural images for 4x upscaling factors and a perceptual loss function which consists of an adversarial loss and a content loss.
The Cityscapes Dataset for Semantic Urban Scene Understanding
11,927 Citations2016Marius Cordts, Mohamed Omran +7 more
This work introduces Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling, and exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity.
Lecture notes in computer sciencePerceptual Losses for Real-Time Style Transfer and Super-Resolution
10,007 Citations2016Justin Johnson, Alexandre Alahi +1 more
This work considers image transformation problems, and proposes the use of perceptual loss functions for training feed-forward networks for image transformation tasks, and shows results on image style transfer, where aFeed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time.
arXiv (Cornell University)Conditional Generative Adversarial Nets
8,874 Citations2014Mehdi Mirza, Simon Osindero
The conditional version of generative adversarial nets is introduced, which can be constructed by simply feeding the data, y, to the generator and discriminator, and it is shown that this model can generate MNIST digits conditioned on class labels.
A Non-Local Algorithm for Image Denoising
6,988 Citations2005Antoni Buades, B. Coll +1 more
A new measure, the method noise, is proposed, to evaluate and compare the performance of digital image denoising methods, and a new algorithm, the nonlocal means (NL-means), based on a nonlocal averaging of all pixels in the image is proposed.
arXiv (Cornell University)Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
6,981 Citations2015Alec Radford, Luke Metz +1 more
Image Style Transfer Using Convolutional Neural Networks
6,010 Citations2016Leon A. Gatys, Alexander S. Ecker +1 more
A Neural Algorithm of Artistic Style is introduced that can separate and recombine the image content and style of natural images and provide new insights into the deep image representations learned by Convolutional Neural Networks and demonstrate their potential for high level image synthesis and manipulation.
arXiv (Cornell University)Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
3,622 Citations2014Liang-Chieh Chen, George Papandreou +3 more
Lecture notes in computer scienceColorful Image Colorization
3,340 Citations2016Richard Zhang, Phillip Isola +1 more
This paper proposes a fully automatic approach to colorization that produces vibrant and realistic colorizations and shows that colorization can be a powerful pretext task for self-supervised feature learning, acting as a cross-channel encoder.
arXiv (Cornell University)Instance Normalization: The Missing Ingredient for Fast Stylization
3,095 Citations2016Dmitry Ulyanov, Andrea Vedaldi +1 more
A small change in the stylization architecture results in a significant qualitative improvement in the generated images, and can be used to train high-performance architectures for real-time image generation.
Holistically-Nested Edge Detection
3,067 Citations2015Saining Xie, Zhuowen Tu
IEEE Computer Graphics and ApplicationsColor transfer between images
2,967 Citations2001Erik Reinhard, M. Adhikhmin +2 more
This work uses a simple statistical analysis to impose one image's color characteristics on another by choosing an appropriate source image and applying its characteristic to another image.
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture
2,855 Citations2015David Eigen, Rob Fergus
This paper addresses three different computer vision tasks using a single basic architecture: depth prediction, surface normal estimation, and semantic labeling using a multiscale convolutional network that is able to adapt easily to each task using only small modifications.
Image quilting for texture synthesis and transfer
2,383 Citations2001Alexei A. Efros, William T. Freeman
This work uses quilting as a fast and very simple texture synthesis algorithm which produces surprisingly good results for a wide range of textures and extends the algorithm to perform texture transfer — rendering an object with a texture taken from a different object.
ACM Transactions on GraphicsRemoving camera shake from a single photograph
1,842 Citations2006Rob Fergus, Barun Singh +3 more
arXiv (Cornell University)Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
1,654 Citations2015Emily Denton, Soumith Chintala +2 more
A generative parametric model capable of producing high quality samples of natural images using a cascade of convolutional networks within a Laplacian pyramid framework to generate images in a coarse-to-fine fashion.
Image analogies
1,552 Citations2001Aaron Hertzmann, Charles Jacobs +3 more
Lecture notes in computer sciencePrecomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks
1,512 Citations2016Chuan Li, Michael Wand
Markovian Generative Adversarial Networks (MGANs) are proposed, a method for training generative networks for efficient texture synthesis that surpasses previous neural texture synthesizers by a significant margin and applies to texture synthesis, style transfer, and video stylization.
arXiv (Cornell University)Improved Techniques for Training GANs
1,386 Citations2016Tim Salimans, Ian Goodfellow +4 more
This work focuses on two applications of GANs: semi-supervised learning, and the generation of images that humans find visually realistic, and presents ImageNet samples with unprecedented resolution and shows that the methods enable the model to learn recognizable features of ImageNet classes.
Lecture notes in computer scienceGenerative Visual Manipulation on the Natural Image Manifold
1,158 Citations2016Jun-Yan Zhu, Philipp Krähenbühl +2 more
This paper proposes to learn the natural image manifold directly from data using a generative adversarial neural network, and defines a class of image editing operations, and constrain their output to lie on that learned manifold at all times.
Multispectral pedestrian detection: Benchmark dataset and baseline
1,127 Citations2015Soonmin Hwang, Jaesik Park +3 more
This dataset introduces multispectral ACF, which is an extension of aggregated channel features (ACF) to simultaneously handle color-thermal image pairs, and achieves another breakthrough in the pedestrian detection task.
MPG.PuRe (Max Planck Society)Generative Adversarial Text to Image Synthesis
1,083 Citations2016Scott Reed, Zeynep Akata +4 more
arXiv (Cornell University)The Cityscapes Dataset for Semantic Urban Scene Understanding
1,000 Citations2024Marius Cordts
Lecture notes in computer scienceLearning Representations for Automatic Colorization
932 Citations2016Gustav Larsson, Michael Maire +1 more
A fully automatic image colorization system that leverages recent advances in deep networks, exploiting both low-level and semantic representations, and explores colorization as a vehicle for self-supervised visual representation learning.
arXiv (Cornell University)Energy-based Generative Adversarial Network
891 Citations2016Junbo Zhao, Michaël Mathieu +1 more
arXiv (Cornell University)Fully Convolutional Networks for Semantic Segmentation
836 Citations2016Jonathan Long, Evan Shelhamer +1 more
ACM Transactions on GraphicsHow do humans sketch objects?
834 Citations2012Mathias Eitz, James Hays +1 more
This paper is the first large scale exploration of human sketches, developing a bag-of-features sketch representation and using multi-class support vector machines, trained on the sketch dataset, to classify sketches.
ACM Transactions on GraphicsLet there be color!
833 Citations2016Satoshi Iizuka, Edgar Simo‐Serra +1 more
A novel technique to automatically colorize grayscale images that combines both global priors and local image features and can process images of any resolution, unlike most existing approaches based on CNN.
International Journal of Computer VisionHolistically-Nested Edge Detection
727 Citations2017Saining Xie, Zhuowen Tu
HED performs image-to-image prediction by means of a deep learning model that leverages fully convolutional neural networks and deeply-supervised nets, and automatically learns rich hierarchical representations that are important in order to resolve the challenging ambiguity in edge and object boundary detection.
Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
696 Citations2016Chuan Li, Michael Wand
A combination of generative Markov random field models and discriminatively trained deep convolutional neural networks for synthesizing 2D images, yielding results far out of reach of classic generative MRF methods.
ACM Transactions on GraphicsWhat makes Paris look like Paris?
687 Citations2012Carl Doersch, Saurabh Singh +3 more
It is shown that geographically representative image elements can be discovered automatically from Google Street View imagery in a discriminative manner and it is demonstrated that these elements are visually interpretable and perceptually geo-informative.
arXiv (Cornell University)Autoencoding beyond pixels using a learned similarity metric
662 Citations2015Anders Larsen, Søren Kaae Sønderby +2 more
Lecture notes in computer scienceGenerative Image Modeling Using Style and Structure Adversarial Networks
635 Citations2016Xiaolong Wang, Abhinav Gupta
This paper factorize the image generation process and proposes Style and Structure Generative Adversarial Network, a model that is interpretable, generates more realistic images and can be used to learn unsupervised RGBD representations.
Fine-Grained Visual Comparisons with Local Learning
490 Citations2014Aron Yu, Kristen Grauman
This work proposes a local learning approach for fine-grained visual comparisons that outperforms state-of-the-art methods for relative attribute prediction and shows how to identify analogous pairs using learned metrics.
arXiv (Cornell University)Texture Synthesis Using Convolutional Neural Networks
424 Citations2015Leon A. Gatys, Alexander S. Ecker +1 more
A new model of natural textures based on the feature spaces of convolutional neural networks optimised for object recognition is introduced, showing that across layers the texture representations increasingly capture the statistical properties of natural images while making object information more and more explicit.
Removing camera shake from a single photograph
377 Citations2006Rob Fergus, Barun Singh +3 more
This work introduces a method to remove the effects of camera shake from seriously blurred images, which assumes a uniform camera blur over the image and negligible in-plane camera rotation.
arXiv (Cornell University)Deep multi-scale video prediction beyond mean square error
367 Citations2015Michaël Mathieu, Camille Couprie +1 more
This work trains a convolutional network to generate future frames given an input sequence and proposes three different and complementary feature learning strategies: a multi-scale architecture, an adversarial training method, and an image gradient difference loss function.
Lecture notes in computer scienceSpatial Pattern Templates for Recognition of Objects with Regular Structure
362 Citations2013Radim Tyleček, Radim Šára
The paper describes how to embody specific spatial relations in a representation called Spatial Pattern Templates (SPT), which allows us to capture regularity constraints of alignment and equal spacing in pairwise and ternary potentials.
ACM Transactions on GraphicsTransient attributes for high-level understanding and editing of outdoor scenes
293 Citations2014Pierre-Yves Laffont, Zhile Ren +3 more
This work studies "transient scene attributes" -- high level properties which affect scene appearance, such as "snow", "autumn", "dusk", "fog", and defines 40 transient attributes and uses crowdsourcing to annotate thousands of images from 101 webcams to train regressors that can predict the presence of attributes in novel images.
ACM Transactions on GraphicsData-driven hallucination of different times of day from a single outdoor photo
202 Citations2013YiChang Shih, Sylvain Paris +2 more
This paper introduces the first data-driven approach to automatically creating a plausible-looking photo that appears as though it were taken at a different time of day, using a database of time-lapse videos of various scenes.
Lecture notes in computer scienceLearning Temporal Transformations from Time-Lapse Videos
144 Citations2016Yipin Zhou, Tamara L. Berg
This paper learns computational models of object transformations from time-lapse videos and explores the use of generative models to create depictions of objects at future times to explore several different prediction tasks.
arXiv (Cornell University)Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
127 Citations2014David Eigen, Rob Fergus
arXiv (Cornell University)Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
116 Citations2016Levent Karacan, Zeynep Akata +2 more
A novel deep conditional generative adversarial network architecture that takes its strength from the semantic layout and scene attributes integrated as conditioning variables and is able to generate realistic outdoor scene images under different conditions, e.g. day-night, sunny-foggy, with clear object boundaries.
Generating Interpretable Images with Controllable Structure
56 Citations2017Scott Reed, Aäron van den Oord +4 more
Improved text-to-image synthesis with controllable object locations using an extension of Pixel Convolutional Neural Networks (PixelCNN) and it is shown how the model can generate images conditioned on part keypoints and segmentation masks.
Visually Indicated Sounds
17 Citations2016Andrew Owens, Phillip Isola +4 more
This paper presents an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick, using a recurrent neural network to predict sound features from videos and then producing a waveform from these features with an example-based synthesis procedure.
arXiv (Cornell University)Top-Down Learning for Structured Labeling with Convolutional Pseudoprior
11 Citations2015Saining Xie, Xun Huang +1 more