Understanding Dataset Design Choices for Multi-hop Reasoning
Published 1 January 2019Open access
Jifan Chen, Greg Durrett
Citations93
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper investigates two recently proposed datasets, WikiHop and HotpotQA, and explores sentence-factored models for these tasks; by design, these models cannot do multi-hop reasoning, but they are still able to solve a large number of examples in both datasets.
Abstract
Jifan Chen, Greg Durrett. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
Keywords
Computer Science
arXiv (Cornell University)Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
10,728 Citations2014Jun‐Young Chung, Çaǧlar Gülçehre +2 more
These advanced recurrent units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU), are found to be comparable to LSTM.
SQuAD: 100,000+ Questions for Machine Comprehension of Text
6,328 Citations2016Pranav Rajpurkar, Jian Zhang +2 more
A strong logistic regression model is built, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%).
Oxford University Research Archive (ORA) (University of Oxford)Teaching Machines to Read and Comprehend
1,936 Citations2015Karl Moritz Hermann, Tomáš Kočiský +5 more
arXiv (Cornell University)Teaching Machines to Read and Comprehend
1,526 Citations2015Karl Moritz Hermann, Tomáš Kočiský +5 more
A new methodology is defined that resolves this bottleneck and provides large scale supervised reading comprehension data that allows a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure to be developed.
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
1,436 Citations2018Zhilin Yang, Peng Qi +5 more
It is shown that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.
arXiv (Cornell University)Bidirectional Attention Flow for Machine Comprehension
1,290 Citations2016Min Joon Seo, Aniruddha Kembhavi +2 more
arXiv (Cornell University)Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
727 Citations2015Jason Weston, Antoine Bordes +5 more
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
668 Citations2013Matthew Richardson, Christopher J. C. Burges +1 more
MCTest is presented, a freely available set of stories and associated questions intended for research on the machine comprehension of text that requires machines to answer multiple-choice reading comprehension questions about fictional stories, directly tackling the high-level goal of open-domain machine comprehension.
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
635 Citations2018Todor Mihaylov, Peter E. Clark +2 more
A new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject, and oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts.
arXiv (Cornell University)Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
631 Citations2015Ankit Kumar, Ozan İrsoy +7 more
The dynamic memory network (DMN), a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers, is introduced.
A Hierarchical Neural Autoencoder for Paragraphs and Documents
518 Citations2015Jiwei Li, Thang Luong +1 more
This paper introduces an LSTM model that hierarchically builds an embedding for a paragraph from embeddings for sentences and words, then decodes this embedding to reconstruct the original paragraph and evaluates the reconstructed paragraph using standard metrics to show that neural models are able to encode texts in a way that preserve syntactic, semantic, and discourse coherence.
Zenodo (CERN European Organization for Nuclear Research)QAngaroo (MedHop + WikiHop) - Constructing Datasets for Multi-hop Reading Comprehension Across Documents
494 Citations2018Johannes Welbl, Pontus Stenetorp +1 more
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
442 Citations2016Danqi Chen, Jason Bolton +1 more
A thorough examination of this new reading comprehension task by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and showing that a neural network can be trained to give good performance on this task.
arXiv (Cornell University)QANet: Combining Local Convolution with Global Self-Attention for\n Reading Comprehension
440 Citations2018Adams Wei Yu, D. Dohan +5 more
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
392 Citations2018Daniel Khashabi, Snigdha Chaturvedi +3 more
The dataset is the first to study multi-sentence inference at scale, with an open-ended set of question types that requires reasoning skills, and finds human solvers to achieve an F1-score of 88.1%.
arXiv (Cornell University)The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
334 Citations2015Felix Hill, Antoine Bordes +2 more
arXiv (Cornell University)The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
307 Citations2016Felix Hill, Antoine Bordes +2 more
There is a sweet-spot, not too big and not too small, between single words and full sentences that allows the most meaningful information in a text to be effectively retained and recalled, and models which store explicit representations of long-term contexts outperform state-of-the-art neural language models at predicting semantic content words.
ReasoNet
265 Citations2017Yelong Shen, Po-Sen Huang +2 more
A novel neural network architecture called the Reasoning Network (ReasoNet) for machine comprehension tasks, which makes use of multiple turns to effectively exploit and then reason over the relation among queries, documents, and answers.
Making Neural QA as Simple as Possible but not Simpler
176 Citations2017Dirk Weissenborn, Georg Wiese +1 more
This work proposes a simple heuristic that guides the development of neural baseline systems for the extractive QA task and finds that there are two ingredients necessary for building a high-performing neural QA system: the awareness of question words while processing the context and a composition function that goes beyond simple bag-of-words modeling, such as recurrent neural networks.
arXiv (Cornell University)Exploring Graph-structured Passage Representation for Multi-hop Reading Comprehension with Graph Neural Networks
85 Citations2018Linfeng Song, Zhiguo Wang +4 more
A new method for better connecting global evidence is introduced, which forms more complex graphs compared to DAGs, and Experiments on two standard datasets show that richer global information leads to better answers.
arXiv (Cornell University)ReasoNet: Learning to Stop Reading in Machine Comprehension
75 Citations2016Yelong Shen, Po-Sen Huang +2 more
arXiv (Cornell University)WorldTree: A Corpus of Explanation Graphs for Elementary Science\n Questions supporting Multi-Hop Inference
32 Citations2018Peter Jansen, Elizabeth Wainwright +2 more