I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling
Published 1 January 2021Open access
Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, Jason Weston
Citations49
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues are introduced and it is shown that the best contradiction detection model correlates well with human judgments and is used in both automatically evaluating and improving the consistency of state-of-the-art generative chatbots.
Abstract
Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, Jason Weston. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Keywords
Computer Science
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case
17,334 Citations2019Yinhan Liu, Myle Ott +8 more
This work considers the task of building machine learning models to automatically select the best combination for a problem instance and contributes to the automatic learning of instance features directly from the high-level representation of a problem instance using a transformer encoder.
Pattern RecognitionThe use of the area under the ROC curve in the evaluation of machine learning algorithms
7,233 Citations1997Andrew P. Bradley
AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests; a standard error that decreased as both AUC and the number of test samples increased; decision threshold independent; and it is invariant to a priori class probabilities.
Journal of Social IssuesMachines and Mindlessness: Social Responses to Computers
3,627 Citations2000Clifford Nass, Youngme Moon
A series of experimental studies demonstrate that individuals mindlessly apply social rules and expectations to computers, and demonstrates that people exhibit overlearned social behaviors such as politeness and reciprocity toward comput-ers.
A large annotated corpus for learning natural language inference
3,452 Citations2015Samuel R. Bowman, Gabor Angeli +2 more
The Stanford Natural Language Inference corpus is introduced, a new, freely available collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning, which allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time.
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
1,725 Citations2016Iulian Vlad Serban, Alessandro Sordoni +3 more
Lecture notes in computer scienceThe PASCAL Recognising Textual Entailment Challenge
1,626 Citations2006Ido Dagan, Oren Glickman +1 more
Personalizing Dialogue Agents: I have a dog, do you have pets too?
1,146 Citations2018Saizheng Zhang, Emily Dinan +4 more
This work collects data and train models tocondition on their given profile information; and information about the person they are talking to, resulting in improved dialogues, as measured by next utterance prediction.
Hierarchical Neural Story Generation
1,124 Citations2018Angela Fan, Mike Lewis +1 more
This work collects a large dataset of 300K human-written stories paired with writing prompts from an online forum that enables hierarchical story generation, where the model first generates a premise, and then transforms it into a passage of text.
A Persona-Based Neural Conversation Model
897 Citations2016Jiwei Li, Michel Galley +4 more
This work presents persona-based models for handling the issue of speaker consistency in neural response generation that yield qualitative performance improvements in both perplexity and BLEU scores over baseline sequence-to-sequence models.
Adversarial NLI: A New Benchmark for Natural Language Understanding
609 Citations2020Yixin Nie, Adina Williams +4 more
This work introduces a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure, and shows that non-expert annotators are successful at finding their weaknesses.
arXiv (Cornell University)Deep Learning: A Critical Appraisal
387 Citations2018Gary Marcus
Ten concerns for deep learning are presented, and it is suggested that deep learning must be supplemented by other techniques if the authors are to reach artificial general intelligence.
The Springer series on challenges in machine learningThe Second Conversational Intelligence Challenge (ConvAI2)
362 Citations2019Emily Dinan, Varvara Logacheva +15 more
To improve performance on multi-turn conversations with humans, future systems must go beyond single word metrics like perplexity to measure the performance across sequences of utterances (conversations)—in terms of repetition, consistency and balance of dialogue acts.
arXiv (Cornell University)Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence\n Models
358 Citations2016Ashwin K. Vijayakumar, Michael Cogswell +5 more
arXiv (Cornell University)Towards a Human-like Open-Domain Chatbot
266 Citations2020Daniel Adiwardana
Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations, is presented and a human evaluation metric called Sensibleness and Specificity Average (SSA) is proposed, which captures key elements of a human-like multi- turn conversation.
Evaluating Models’ Local Decision Boundaries via Contrast Sets
258 Citations2020Matt Gardner, Yoav Artzi +24 more
A more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data, and recommends that the dataset authors manually perturb the test instances in small but meaningful ways that (typically) change the gold label, creating contrast sets.
Dialogue Natural Language Inference
247 Citations2019Sean Welleck, Jason Weston +2 more
This paper proposes a method which demonstrates that a model trained on Dialogue NLI can be used to improve the consistency of a dialogue model, and evaluates the method with human evaluation and with automatic metrics on a suite of evaluation sets designed to measure a dialoguemodel’s consistency.
arXiv (Cornell University)Wizard of Wikipedia: Knowledge-Powered Conversational Agents
245 Citations2018Emily Dinan, Stephen Roller +4 more
The best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while a new benchmark allows for measuring further improvements in this important research direction.
What makes a good conversation? How controllable attributes affect human judgments
226 Citations2019Abigail See, Stephen Roller +2 more
This work examines two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chit-chat dialogue: repetition, specificity, response-relatedness and question-asking, and shows that by controlling combinations of these variables their models obtain clear improvements in human quality judgments.
Retrieve and Refine: Improved Sequence Generation Models For Dialogue
176 Citations2018Jason Weston, Emily Dinan +1 more
This work develops a model that combines the two approaches to avoid both their deficiencies: first retrieve a response and then refine it – the final sequence generator treating the retrieval as additional context.
Recipes for Building an Open-Domain Chatbot
171 Citations2021Stephen Roller, Emily Dinan +9 more
Human evaluations show the best models outperform existing approaches in multi-turn dialogue on engagingness and humanness measurements, and the limitations of this work are discussed by analyzing failure cases of the models.
Zenodo (CERN European Organization for Nuclear Research)The Pushshift Reddit Dataset
161 Citations2020Jason Baumgartner, Savvas Zannettou +3 more
The Pushshift Reddit dataset makes it possible for social media researchers to reduce time spent in the data collection, cleaning, and storage phases of their projects.
Assigning Personality/Profile to a Chatting Machine for Coherent Conversation Generation
155 Citations2018Qian Qiao, Minlie Huang +3 more
Manual and automatic evaluation shows that the model can deliver more coherent, natural, and diversified responses to profile-coherent responses.
arXiv (Cornell University)Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
154 Citations2019Samuel Humeau, Kurt Shuster +2 more
This work develops a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features and achieves state-of-the-art results on three existing tasks.
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
120 Citations2020Margaret Li, Stephen Roller +5 more
This work shows how all of the problems of generative dialogue models can be addressed by extending the recently introduced unlikelihood loss to these cases, and demonstrates the efficacy of this approach across several dialogue tasks.
World Wide WebNeural personalized response generation as domain adaptation
91 Citations2018Weinan Zhang, Qingfu Zhu +3 more
Experimental results show that the proposed personalized response generation model outperforms the state-of-the-art approaches to language model personalization and persona-based neural conversation generation on the automatic evaluation, offline human judgment and the quasi-Turing test.
arXiv (Cornell University)ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
79 Citations2019Margaret Li, Jason Weston +1 more
A novel procedure involving comparing two full dialogues, where a human judge is asked to pay attention to only one speaker within each, and make a pairwise judgment, resulting in better tests.
What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
77 Citations2020Yixin Nie, Xiang Zhou +1 more
This work collects ChaosNLI, a dataset with a total of 464,500 annotations to study Collective HumAn OpinionS in oft-used NLI evaluation sets, and argues for a detailed examination of human agreement in future data collection efforts, and evaluating model outputs against the distribution over collective human opinions.
Steering Output Style and Topic in Neural Response Generation
66 Citations2017Di Wang, Nebojša Jojić +2 more
The decompose the neural generation process into empirically easier sub-problems: a faithfulness model and a decoding method based on selective-sampling and training and sampling algorithms that bias the generation process with a specific language style restriction, or a topic restriction.
arXiv (Cornell University)Consistent Dialogue Generation with Self-supervised Feature Learning
31 Citations2019Yizhe Zhang, Xiang Gao +5 more
This paper proposes a neural conversation model that generates consistent responses by maintaining certain features related to topics and personas throughout the conversation by adopting a binary feature representation and introducing a feature disentangling loss.