Prefix-Tuning: Optimizing Continuous Prompts for Generation
Published 1 January 2021Open access
Xiang Lisa Li, Percy Liang
Citations2,205
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Prefix-tuning is proposed, a lightweight alternative to fine- Tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which is called the prefix.
Abstract
Xiang Lisa Li, Percy Liang. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Keywords
Computer Science
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case
17,334 Citations2019Yinhan Liu, Myle Ott +8 more
This work considers the task of building machine learning models to automatically select the best combination for a problem instance and contributes to the automatic learning of instance features directly from the high-level representation of a problem instance using a transformer encoder.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics
14,235 Citations2023T. B. Brown, Eliina Kaugeranna +1 more
arXiv (Cornell University)Decoupled Weight Decay Regularization
9,113 Citations2017Ilya Loshchilov, Frank Hutter
This work proposes a simple modification to recover the original formulation of weight decay regularization by decoupling the weight decay from the optimization steps taken w.r.t. the loss function, and provides empirical evidence that this modification substantially improves Adam's generalization performance.
ROUGE: A Package for Automatic Evaluation of Summaries
8,286 Citations2004Chin-Yew Lin
Four different RouGE measures are introduced: ROUGE-N, ROUge-L, R OUGE-W, and ROUAGE-S included in the Rouge summarization evaluation package and their evaluations.
Transformers: State-of-the-Art Natural Language Processing
8,019 Citations2020Thomas Wolf, Lysandre Debut +20 more
The \textit{Transformers} library is an open-source library that consists of carefully engineered state-of-the art Transformer architectures under a unified API and a curated collection of pretrained models made by and available for the community.
CIDEr: Consensus-based image description evaluation
4,724 Citations2015Ramakrishna Vedantam, C. Lawrence Zitnick +1 more
A novel paradigm for evaluating image descriptions that uses human consensus is proposed and a new automated metric that captures human judgment of consensus better than existing metrics across sentences generated by various sources is evaluated.
A Study of Translation Edit Rate with Targeted Human Annotation
2,402 Citations2006Matthew Snover, Bonnie J. Dorr +3 more
A new, intuitive measure for evaluating machine translation output that avoids the knowledge intensiveness of more meaning-based approaches, and the labor-intensiveness of human judgments is defined.
Privacy-Preserving Deep Learning
2,215 Citations2015Reza Shokri, Vitaly Shmatikov
This paper presents a practical system that enables multiple parties to jointly learn an accurate neural-network model for a given objective without sharing their input datasets, and exploits the fact that the optimization algorithms used in modern deep learning, namely, those based on stochastic gradient descent, can be parallelized and executed asynchronously.
arXiv (Cornell University)BERTScore: Evaluating Text Generation with BERT
2,045 Citations2019Tianyi Zhang, Varsha Kishore +3 more
Text Summarization with Pretrained Encoders
1,572 Citations2019Yang Liu, Mirella Lapata
This paper introduces a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences and proposes a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two.
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
1,197 Citations2018Shashi Narayan, Shay B. Cohen +1 more
A novel abstractive model is proposed which is conditioned on the article’s topics and based entirely on convolutional neural networks, outperforming an oracle extractive system and state-of-the-art abstractive approaches when evaluated automatically and by humans.
Zenodo (CERN European Organization for Nuclear Research)Transformers: State-of-the-Art Natural Language Processing
1,172 Citations2020Thomas Wolf, Lysandre Debut +14 more
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
1,114 Citations2020Taylor Shin, Yasaman Razeghi +3 more
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
1,055 Citations2020Yizhe Zhang, Siqi Sun +7 more
It is shown that conversational systems that leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline systems.
Meteor
1,045 Citations2007Alon Lavie, Abhaya Agarwal
The technical details underlying the Meteor metric are recapped, the latest release includes improved metric parameters and extends the metric to support evaluation of MT output in Spanish, French and German, in addition to English.
arXiv (Cornell University)Federated Learning of Deep Networks using Model Averaging
984 Citations2016H. Brendan McMahan, Eider Moore +2 more
This work presents a practical method for the federated learning of deep networks that proves robust to the unbalanced and non-IID data distributions that naturally arise, and allows high-quality models to be trained in relatively few rounds of communication.
arXiv (Cornell University)CTRL: A Conditional Transformer Language Model for Controllable Generation
786 Citations2019Nitish Shirish Keskar, Bryan McCann +3 more
CTRL is released, a 1.63 billion-parameter conditional transformer language model, trained to condition on control codes that govern style, content, and task-specific behavior, providing more explicit control over text generation.
arXiv (Cornell University)Multilingual Denoising Pre-training for Neural Machine Translation
607 Citations2020Yinhan Liu, Jiatao Gu +6 more
arXiv (Cornell University)BERTScore: Evaluating Text Generation with BERT
603 Citations2020Tianyi Zhang, Varsha Kishore +3 more
This work proposes BERTScore, an automatic evaluation metric for text generation that correlates better with human judgments and provides stronger model selection performance than existing metrics.
Oxford University Research Archive (ORA) (University of Oxford)Learning multiple visual domains with residual adapters
495 Citations2017Sylvestre-Alvise Rebuffi, Hakan Bilen +1 more
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
465 Citations2019Wei Zhao, Maxime Peyrard +4 more
This paper investigates strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality and validate the new metric, namely MoverScore, on a number of text generation tasks.
Extractive Summarization as Text Matching
412 Citations2020Ming Zhong, Pengfei Liu +4 more
This paper forms the extractive summarization task as a semantic text matching problem, in which a source document and candidate summaries will be matched in a semantic space to create a semantic matching framework.
arXiv (Cornell University)Plug and Play Language Models: A Simple Approach to Controlled Text\n Generation
407 Citations2019Sumanth Dathathri, Andrea Madotto +6 more
The Plug and Play Language Model (PPLM) for controllable language generation is proposed, which combines a pretrained LM with one or more simple attribute classifiers that guide text generation without any further training of the LM.
The WebNLG Challenge: Generating Text from RDF Data
334 Citations2017Claire Gardent, Anastasia Shimorina +2 more
The microplanning task is introduced, data preparation, evaluation methodology, participant results and a brief description of the participating systems are provided.
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
317 Citations2021Guanghui Qin, Jason Eisner
This work explores the idea of learning prompts by gradient descent—either fine-tuning prompts taken from previous work, or starting from random initialization, showing that the implicit factual knowledge in language models was previously underestimated.
GeDi: Generative Discriminator Guided Sequence Generation
209 Citations2021Ben Krause, Akhilesh Deepak Gotmare +5 more
GeDi is proposed as an efficient method for using smaller LMs as generative discriminators to guide generation from large LMs to make them safer and more controllable, and is found that GeDi gives stronger controllability than the state of the art method while also achieving generation speeds more than 30 times faster.
Incorporating BERT into Neural Machine Translation
208 Citations2020Jinhua Zhu, Yingce Xia +6 more
A new algorithm named BERT-fused model is proposed, in which BERT is first used to extract representations for an input sequence, and then the representations are fused with each layer of the encoder and decoder of the NMT model through attention mechanisms.
Research Portal (King's College London)Comparing automatic and human evaluation of NLG systems
179 Citations2006Anja Belz, Ehud Reiter
It is found that NI ST scores correlate best with human judgments, but that all automatic metrics the authors examined are biased in favour of generators that select on the basis of frequency alone.
arXiv (Cornell University)Incorporating BERT into Neural Machine Translation
172 Citations2020Jinhua Zhu, Yingce Xia +6 more
Proceedings of the 2021 Conference on Empirical Methods in Natural Language ProcessingThe Power of Scale for Parameter-Efficient Prompt Tuning
94 Citations2021Brian Lester, Rami Al‐Rfou +1 more
This work explores “prompt tuning,” a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks and shows that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient “Prompt ensembling.”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
91 Citations2020Zhaojiang Lin, Andrea Madotto +1 more
The experiments show that by just using an additional 2-3% parameters for each task, the model can maintain or even improve the performance of fine-tuning the whole model.
DART: Open-Domain Structured Data Record to Text Generation
76 Citations2021Linyong Nan, Dragomir Radev +22 more
The dataset construction framework effectively merged heterogeneous sources from open domain semantic parsing and spoken dialogue systems by utilizing techniques including tree ontology annotation, question-answer pair to declarative sentence conversion, and predicate unification, all with minimum post-editing.
AI OpenGPT understands, too
75 Citations2023Xiao Liu, Yanan Zheng +5 more
A novel method P-Tuning is proposed that employs trainable continuous prompt embeddings in concatenation with discrete prompts that stabilizes training by minimizing the gap between various discrete prompts, and improves performance by a sizeable margin on a wide range of NLU tasks including LAMA and SuperGLUE.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
74 Citations2021Timo Schick, Hinrich Schütze
This work introduces Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task.
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models
53 Citations2020Mengjie Zhao, Tao Lin +3 more
Analyzing the loss landscape, it is shown that Masking and finetuning produce models that reside in minima that can be connected by a line segment with nearly constant test accuracy, confirming that masking can be utilized as an efficient alternative to finetuned.
Pragmatically Informative Text Generation
49 Citations2019Sheng Shen, Daniel Fried +2 more
This work considers two pragmatic modeling methods for text generation: one where pragmatics is imposed by information preservation, and another where prag matics isimposed by explicit modeling of distractors.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
41 Citations2021Jonas Pfeiffer, Aishwarya Kamath +3 more
This work proposes AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks by separating the two stages, i.e., knowledge extraction and knowledge composition, so that the classifier can effectively exploit the representations learned frommultiple tasks in a non-destructive manner.
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
30 Citations2021Armen Aghajanyan, Luke Zettlemoyer +1 more
This paper empirically shows that common pre-trained models have a very low intrinsic dimension, and connects intrinsic dimensionality with low dimensional task representations and compression based generalization bounds to provide intrinsic-dimension-based generalizations bounds that are independent of the full parameter count.
arXiv (Cornell University)How fine can fine-tuning be? Learning efficient language models
26 Citations2020Evani Radiya-Dixit, Xin Wang
arXiv (Cornell University)DART: Open-Domain Structured Data Record to Text Generation
24 Citations2020Dragomir Radev, Rui Zhang +21 more
arXiv (Cornell University)Masking as an Efficient Alternative to Finetuning for Pretrained Language Models
13 Citations2020Mengjie Zhao, Tao Lin +3 more
arXiv (Cornell University)Can Unconditional Language Models Recover Arbitrary Sentences
13 Citations2019Nishant Subramani, Samuel R. Bowman +1 more
Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
6 Citations2021Asa Cooper Stickland, Xian Li +1 more
The benefits and drawbacks of freezing parameters, and adding new ones, when fine-tuning a pre-trained model on Machine Translation (MT), are investigated.
arXiv (Cornell University)Can Unconditional Language Models Recover Arbitrary Sentences?
4 Citations2019Nishant Subramani, Samuel R. Bowman +1 more
This work introduces a pair of effective complementary methods for feeding representations into pretrained unconditional language models and a corresponding set of methods to map sentences into and out of this representation space, the \textit{reparametrized sentence space}.