CTRL: A Conditional Transformer Language Model for Controllable Generation
arXiv (Cornell University)Published 11 September 2019Open access
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, Richard Socher
Citations786
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
CTRL is released, a 1.63 billion-parameter conditional transformer language model, trained to condition on control codes that govern style, content, and task-specific behavior, providing more explicit control over text generation.
Abstract
Large-scale language models show promising text generation capabilities, but users cannot easily control particular aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model, trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data via model-based source attribution. We have released multiple full-sized, pretrained versions of CTRL at https://github.com/salesforce/ctrl.
Keywords
Computer Science
Deep Residual Learning for Image Recognition
222,082 Citations2016Kaiming He, Xiangyu Zhang +2 more
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
UvA-DARE (University of Amsterdam)Adam: A Method for Stochastic Optimization
84,783 Citations2014Diederik P. Kingma, Jimmy Ba
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
Journal of Japan Society for Fuzzy Theory and Intelligent InformaticsGAN（Generative Adversarial Nets）
21,715 Citations2017Ian Goodfellow
arXiv (Cornell University)Distributed Representations of Words and Phrases and their Compositionality
18,086 Citations2013Tomáš Mikolov, Ilya Sutskever +3 more
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
International Conference on Machine LearningRectified Linear Units Improve Restricted Boltzmann Machines
13,244 Citations2010Vinod Nair, Geoffrey E. Hinton
Restricted Boltzmann machines were developed using binary stochastic hidden units that learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset.
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
8,612 Citations2011John C. Duchi, Elad Hazan +1 more
arXiv (Cornell University)Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
5,634 Citations2016Yonghui Wu, Mike Schuster +29 more
GNMT, Google's Neural Machine Translation system, is presented, which attempts to address many of the weaknesses of conventional phrase-based translation systems and provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delicited models.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
Get To The Point: Summarization with Pointer-Generator Networks
3,813 Citations2017Abigail See, Peter J. Liu +1 more
A novel architecture that augments the standard sequence-to-sequence attentional model in two orthogonal ways, using a hybrid pointer-generator network that can copy words from the source text via pointing, which aids accurate reproduction of information, while retaining the ability to produce novel words through the generator.
Research PolicyDeveloping a framework for responsible innovation
2,814 Citations2013Jack Stilgoe, Richard Owen +1 more
LinguisticsSometimes I’ll start a sentence in Spanish Y TERMINO EN ESPAÑOL: toward a typology of code-switching1
2,451 Citations1980Shana Poplack
arXiv (Cornell University)InfoGAN: Interpretable Representation Learning by Information Maximizing\n Generative Adversarial Nets
2,414 Citations2016Xi Chen, Yan Duan +4 more
Image-Based Recommendations on Styles and Substitutes
2,215 Citations2015Julian McAuley, Christopher Targett +2 more
The approach is not based on fine-grained modeling of user annotations but rather on capturing the largest dataset possible and developing a scalable method for uncovering human notions of the visual relationships within.
Transactions of the Association for Computational LinguisticsNatural Questions: A Benchmark for Question Answering Research
1,994 Citations2019Tom Kwiatkowski, Jennimaria Palomaki +16 more
The Natural Questions corpus, a question answering data set, is presented, introducing robust metrics for the purposes of evaluating question answering systems; demonstrating high human upper bounds on these metrics; and establishing baseline results using competitive methods drawn from related literature.
International Conference on Learning RepresentationsGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
1,865 Citations2018Alex Wang, Amanpreet Singh +4 more
Deep Contextualized Word Representations
1,785 Citations2018Matthew E. Peters, Mark E Neumann +5 more
A new type of deep contextualized word representation is introduced that models both complex characteristics of word use and how these uses vary across linguistic contexts, allowing downstream models to mix different types of semi-supervision signals.
Applied Physics Letters10.1162/153244303322533223
1,771 Citations2000
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences.
Transactions of the Association for Computational LinguisticsGoogle’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
1,742 Citations2017Melvin Johnson, Mike Schuster +10 more
This work proposes a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages using a shared wordpiece vocabulary, and introduces an artificial token at the beginning of the input sentence to specify the required target language.
arXiv (Cornell University)Cross-lingual Language Model Pretraining
1,616 Citations2019Guillaume Lample, Alexis Conneau
This work proposes two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingsual language model objective.
arXiv (Cornell University)Teaching Machines to Read and Comprehend
1,526 Citations2015Karl Moritz Hermann, Tomáš Kočiský +5 more
A new methodology is defined that resolves this bottleneck and provides large scale supervised reading comprehension data that allows a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure to be developed.
arXiv (Cornell University)The Curious Case of Neural Text Degeneration
1,096 Citations2019Ari Holtzman, Jan Buys +3 more
The Winograd Schema Challenge
859 Citations2011Hector J. Levesque
An alternative to the Turing Test that has some conceptual and practical advantages is presented, and a collection of Winograd schemas is compiled so that the correct answer is obvious to the human reader, but cannot be found using selectional restrictions or statistical techniques over text corpora.
arXiv (Cornell University)SQuAD: 100,000+ Questions for Machine Comprehension of Text
819 Citations2016Pranav Rajpurkar, Jian Zhang +2 more
LanguageConstraints on Language Mixing: Intrasentential Code-Switching and Borrowing in Spanish/English
777 Citations1979Carol W. Pfaff
A Neural Attention Model for Abstractive Sentence Summarization
703 Citations2015Alexander M. Rush, Sumit Chopra +1 more
This work proposes a fully data-driven approach to abstractive sentence summarization by utilizing a local attention-based model that generates each word of the summary conditioned on the input sentence.
arXiv (Cornell University)Bootstrapping a Data-Set and Model for Question-Answering in Portuguese (Short Paper)
586 Citations2021Zihang Dai, Zhilin Yang +1 more
An exploratory work carried out to come up with a state-of-the-art model for question-answering tasks, for the Portuguese language, based on deep neural networks, achieves 50% accuracy in the testing data-set.
Large Language Models in Machine Translation
550 Citations2007Thorsten Brants, Ashok C. Popat +3 more
Systems, methods, and computer program products for machine translation are provided for backoff score determination as a function of a backoff factor and a relative frequency of a corresponding backoff n-gram in the corpus.
Apollo (University of Cambridge)The malicious use of artificial intelligence: Forecasting, prevention, and mitigation
524 Citations2018Miles Brundage, Shahar Avin +24 more
The following organisations are named on the report: Future of Humanity Institute, University of Oxford, Centre for the Study of Existential Risk, Universityof Cambridge, Center for a New American Security, Electronic Frontier Foundation, OpenAI.
arXiv (Cornell University)Semi-supervised Sequence Learning
490 Citations2015Andrew M. Dai, Quoc V. Le
Two approaches to use unlabeled data to improve Sequence Learning with recurrent networks are presented and it is found that long short term memory recurrent networks after pretrained with the two approaches become more stable to train and generalize better.
arXiv (Cornell University)Generating Long Sequences with Sparse Transformers
488 Citations2019Rewon Child, Scott Gray +2 more
This paper introduces sparse factorizations of the attention matrix which reduce this to $O(n)$, and generates unconditional samples that demonstrate global coherence and great diversity, and shows it is possible in principle to use self-attention to model sequences of length one million or more.
Findings of the 2019 Conference on Machine Translation (WMT19)
475 Citations2019Loïc Barrault, Ondřej Bojar +13 more
This paper presents the results of the premier shared task organized alongside the Conference on Machine Translation (WMT) 2019, asked to build machine translation systems for any of 18 language pairs, to be evaluated on a test set of news stories.
arXiv (Cornell University)Regularizing and Optimizing LSTM Language Models
468 Citations2017Stephen Merity, Nitish Shirish Keskar +1 more
Neural Machine Translation of Rare Words with Subword Units
446 Citations2016Rico Sennrich, Barry Haddow +1 more
This paper introduces a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units, and empirically shows that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.3 BLEU.
arXiv (Cornell University)SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
384 Citations2017Matthew Dunn, Levent Sagun +4 more
It is shown that there is a meaningful gap between the human and machine performances, which suggests that the proposed dataset could well serve as a benchmark for question-answering.
arXiv (Cornell University)Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
375 Citations2016Ramesh Nallapati, Bowen Zhou +3 more
arXiv (Cornell University)Fine-tuned Language Models for Text Classification.
340 Citations2018Jeremy Howard, Sebastian Ruder
Fine-tuned Language Models (FitLaM) is proposed, an effective transfer learning method that can be applied to any task in NLP, and techniques that are key for fine-tuning a state-of-the-art language model are introduced.
arXiv (Cornell University)The Natural Language Decathlon: Multitask Learning as Question Answering
338 Citations2018Bryan McCann, Nitish Shirish Keskar +2 more
Presented on August 28, 2018 at 12:15 p.m. in the Pettit Microelectronics Research Center, Room 102 A/B.
arXiv (Cornell University)A Simple Method for Commonsense Reasoning
314 Citations2018Trieu H. Trinh, Quoc V. Le
Key to this method is the use of language models, trained on a massive amount of unlabled data, to score multiple choice questions posed by commonsense reasoning tests, which outperform previous state-of-the-art methods by a large margin.
arXiv (Cornell University)One Model To Learn Them All
257 Citations2017Łukasz Kaiser, Aidan N. Gomez +5 more
It is shown that tasks with less data benefit largely from joint training with other tasks, while performance on large tasks degrades only slightly if at all, and that adding a block to the model never hurts performance and in most cases improves it on all tasks.
arXiv (Cornell University)Neural Text Generation with Unlikelihood Training
241 Citations2019Sean Welleck, Ilia Kulikov +4 more
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
212 Citations2017Mandar Joshi, Eunsol Choi +2 more
It is shown that, in comparison to other recently introduced large-scale datasets, TriviaQA has relatively complex, compositional questions, has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and requires more cross sentence reasoning to find answers.
arXiv (Cornell University)Fast Decoding in Sequence Models using Discrete Latent Variables
177 Citations2018Łukasz Kaiser, Aurko Roy +5 more
arXiv (Cornell University)Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
172 Citations2016Hakan Inan, Khashayar Khosravi +1 more
This work introduces a novel theoretical framework that facilitates better learning in language modeling, and shows that this framework leads to tying together the input embedding and the output projection matrices, greatly reducing the number of trainable variables.
arXiv (Cornell University)Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
163 Citations2018Noam Shazeer, Mitchell Stern
arXiv (Cornell University)Hierarchical Neural Story Generation
151 Citations2018Angela Fan, Mike Lewis +1 more
Unsupervised Question Answering by Cloze Translation
136 Citations2019Patrick Lewis, Ludovic Denoyer +1 more
It is found that modern QA models can learn to answer human questions surprisingly well using only synthetic training data, and is demonstrated that, without using the SQuAD training data at all, this approach achieves 56.4 F1 on SQuad v1.
arXiv (Cornell University)HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
111 Citations2018Zhilin Yang, Peng Qi +5 more
arXiv (Cornell University)Unsupervised Neural Machine Translation
95 Citations2017Mikel Artetxe, Gorka Labaka +2 more
Synthese Library/Synthese libraryArtificial Intelligence and Responsible Innovation
92 Citations2016Miles Brundage
It is argued that work on the societal dimensions of AI can be enriched by engagement with the literature on “responsible innovation,” which has up until now focused on technical domains like nanotechnology, synthetic biology, and geoengineering.
Using the Output Embedding to Improve Language Models
91 Citations2017Ofir Press, Lior Wolf
The topmost weight matrix of neural network language models is studied and it is shown that this matrix constitutes a valid word embedding and a new method of regularizing the output embedding is offered.
arXiv (Cornell University)Defending Against Neural Fake News
89 Citations2019Rowan Zellers, Ari Holtzman +5 more
A model for controllable text generation called Grover, found that best current discriminators can classify neural fake news from real, human-written, news with 73% accuracy, assuming access to a moderate level of training data, and the best defense against Grover turns out to be Grover itself, with 92% accuracy.
arXiv (Cornell University)Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
87 Citations2019Boris Ginsburg, Patrice Castonguay +8 more
NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay, performs on par or better than well tuned SGD with momentum and Adam or AdamW in experiments on neural networks.
arXiv (Cornell University)NewsQA: A Machine Comprehension Dataset
76 Citations2016Adam Trischler, Tong Wang +5 more
Explain Yourself! Leveraging Language Models for Commonsense Reasoning
59 Citations2019Nazneen Fatema Rajani, Bryan McCann +2 more
This work collects human explanations for commonsense reasoning in the form of natural language sequences and highlighted annotations in a new dataset called Common Sense Explanations to train language models to automatically generate explanations that can be used during training and inference in a novel Commonsense Auto-Generated Explanation framework.
IBM Journal of Research and DevelopmentFactSheets: Increasing trust in AI services through supplier's declarations of conformity
59 Citations2019Matthew Arnold, Rachel Bellamy +12 more
FactSheets is proposed, a comprehensive set of declaration items tailored to AI to help increase trust in AI services that contain purpose, performance, safety, security, and provenance information to be completed by AI service providers for examination by consumers.
arXiv (Cornell University)A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
52 Citations2016Kazuma Hashimoto, Caiming Xiong +2 more
Filling Gender & Number Gaps in Neural Machine Translation with Black-box Context Injection
42 Citations2019Amit Moryossef, Roee Aharoni +1 more
This work proposes a black-box approach for injecting the missing information to a pre-trained neural machine translation system, allowing to control the morphological variations in the generated translations without changing the underlying model or training data.
arXiv (Cornell University)Unifying Question Answering, Text Classification, and Regression via Span Extraction
28 Citations2019Nitish Shirish Keskar, Bryan McCann +2 more
A unified, span-extraction approach leads to superior or comparable performance in supplementary supervised pre-trained, low-data, and multi-task learning experiments on several question answering, text classification, and regression benchmarks.
Neural Text Summarization: A Critical Evaluation
23 Citations2019Wojciech Kryściński, Nitish Shirish Keskar +3 more
This work critically evaluate key ingredients of the current research setup: datasets, evaluation metrics, and models, and highlights three primary shortcomings: automatically collected datasets leave the task underconstrained and may contain noise detrimental to training and evaluation.
arXiv (Cornell University)Pretrained AI Models: Performativity, Mobility, and Change
19 Citations2019Lav R. Varshney, Nitish Shirish Keskar +1 more
It is discussed how pretrained models move through actor networks as a kind of computationally immutable mobile, but that users also act as agents of technological change by reinterpreting them via fine-tuning and transfer and how this sociological understanding of Pretrained models can inform AI governance frameworks for fairness, accountability, and transparency.
arXiv (Cornell University)Memory-Efficient Adaptive Optimization
19 Citations2019Rohan Anil, Vineet Gupta +2 more
This work describes a novel, simple, and flexible adaptive optimization method with sublinear memory cost that retains the benefits of per-parameter adaptivity while allowing for larger models and mini-batches and gives convergence guarantees for the method.
arXiv (Cornell University)Domain Control for Neural Machine Translation
16 Citations2016Catherine Kobus, Josep Crego +1 more
arXiv (Cornell University)ELI5: Long Form Question Answering
14 Citations2019Angela Fan, Yacine Jernite +4 more
NatureSelf-censorship is not enough
11 Citations2012David Kaiser, Jonathan D. Moreno
The debate over publishing potentially dangerous research on flu viruses would benefit from a closer look at history, argue David Kaiser and Jonathan D. Moreno.
Memory-Efficient Adaptive Optimization for Large-Scale Learning.
11 Citations2019Rohan Anil, Vineet Gupta +2 more
arXiv (Cornell University)Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
10 Citations2019Thomas Scialom, Sylvain Lamprier +2 more
Tagged Back-Translation
9 Citations2019Isaac Caswell, Ciprian Chelba +1 more
This work proposes a simpler alternative to noising techniques, consisting of tagging back-translated source sentences with an extra token, which outperform noised back-translation in English-Romanian and match performance on English-German, redefining the state-of-the-art on the former.
arXiv (Cornell University)Large Memory Layers with Product Keys
7 Citations2019Guillaume Lample, Alexandre Sablayrolles +3 more
A structured memory which can be easily integrated into a neural network and significantly increases the capacity of the architecture, by up to a billion parameters with a negligible computational overhead is introduced.
arXiv (Cornell University)SumQE: a BERT-based Summary Quality Estimation Model
4 Citations2019Stratos Xenouleas, Prodromos Malakasiotis +2 more
…