Natural Questions: A Benchmark for Question Answering Research
Transactions of the Association for Computational LinguisticsPublished 2 August 2019Open access
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti
Citations1,994
SJR quartileQ1
SJR score1.82
SNIP4.13
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The Natural Questions corpus, a question answering data set, is presented, introducing robust metrics for the purposes of evaluating question answering systems; demonstrating high human upper bounds on these metrics; and establishing baseline results using competitive methods drawn from related literature.
Abstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Keywords
Computer Science
SQuAD: 100,000+ Questions for Machine Comprehension of Text
6,328 Citations2016Pranav Rajpurkar, Jian Zhang +2 more
A strong logistic regression model is built, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%).
A large annotated corpus for learning natural language inference
3,452 Citations2015Samuel R. Bowman, Gabor Angeli +2 more
The Stanford Natural Language Inference corpus is introduced, a new, freely available collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning, which allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time.
Automatic acquisition of hyponyms from large text corpora
3,283 Citations1992Marti A. Hearst
A set of lexico-syntactic patterns that are easily recognizable, that occur frequently and across text genre boundaries, and that indisputably indicate the lexical relation of interest are identified.
Stochastic modelling and applied probabilityA Probabilistic Theory of Pattern Recognition
3,277 Citations1996Luc Devroye, László Györfi +1 more
The Bayes Error and Vapnik-Chervonenkis theory are applied as guide for empirical classifier selection on the basis of explicit specification and explicit enforcement of the maximum likelihood principle.
Know What You Don’t Know: Unanswerable Questions for SQuAD
2,164 Citations2018Pranav Rajpurkar, Robin Jia +1 more
SQuadRUn is a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.
Oxford University Research Archive (ORA) (University of Oxford)Teaching Machines to Read and Comprehend
1,936 Citations2015Karl Moritz Hermann, Tomáš Kočiský +5 more
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
1,597 Citations2002Pierre Isabelle
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
1,436 Citations2018Zhilin Yang, Peng Qi +5 more
It is shown that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.
Reading Wikipedia to Answer Open-Domain Questions
1,435 Citations2017Danqi Chen, Adam Fisch +2 more
This approach combines a search component based on bigram hashing and TF-IDF matching with a multi-layer recurrent neural network model trained to detect answers in Wikipedia paragraphs, indicating that both modules are highly competitive with respect to existing counterparts.
A Decomposable Attention Model for Natural Language Inference
1,372 Citations2016Ankur P. Parikh, Oscar Täckström +2 more
This work proposes a simple neural architecture for natural language inference that uses attention to decompose the problem into subproblems that can be solved separately, thus making it trivially parallelizable.
arXiv (Cornell University)Analysis of Points of Interests Recommended for Leisure Walk Descriptions
1,291 Citations2024Payal Bajaj, Bui, Thi Minh Hoai +2 more
The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering.
Adversarial Examples for Evaluating Reading Comprehension Systems
1,268 Citations2017Robin Jia, Percy Liang
This work proposes an adversarial evaluation scheme for the Stanford Question Answering Dataset that tests whether systems can answer questions about paragraphs that contain adversarially inserted sentences without changing the correct answer or misleading humans.
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing
1,264 Citations2013
EMNLP 2010 received 500 submissions, a new record for the conference, and the program committee was able to accept 125 papers in total (an acceptance rate of 25%).
WikiQA: A Challenge Dataset for Open-Domain Question Answering
857 Citations2015Yi Yang, Wen-tau Yih +1 more
The WIKIQA dataset is described, a new publicly available set of question and sentence pairs, collected and annotated for research on open-domain question answering, which is more than an order of magnitude larger than the previous dataset.
QuAC: Question Answering in Context
686 Citations2018Eunsol Choi, He He +6 more
QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as it shows in a detailed qualitative evaluation.
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
668 Citations2013Matthew Richardson, Christopher J. C. Burges +1 more
MCTest is presented, a freely available set of stories and associated questions intended for research on the machine comprehension of text that requires machines to answer multiple-choice reading comprehension questions about fictional stories, directly tackling the high-level goal of open-domain machine comprehension.
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
635 Citations2018Todor Mihaylov, Peter E. Clark +2 more
A new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject, and oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts.
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
442 Citations2016Danqi Chen, Jason Bolton +1 more
A thorough examination of this new reading comprehension task by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and showing that a neural network can be trained to give good performance on this task.
Neural Information Processing SystemsMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
440 Citations2016Tri Gia Nguyen, Mir Rosenberg +5 more
This new dataset is aimed to overcome a number of well-known weaknesses of previous publicly available datasets for the same task of reading comprehension and question answering, and is the most comprehensive real-world dataset of its kind in both quantity and quality.
arXiv (Cornell University)The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
307 Citations2016Felix Hill, Antoine Bordes +2 more
There is a sweet-spot, not too big and not too small, between single words and full sentences that allows the most meaningful information in a text to be effectively retained and recalled, and models which store explicit representations of long-term contexts outperform state-of-the-art neural language models at predicting semantic content words.
DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
232 Citations2018Wei He, Kai Liu +11 more
This paper introduces DuReader, a new large-scale, open-domain Chinese machine reading comprehension (MRC) dataset, designed to address real-world MRC, and organizes a shared competition to encourage the exploration of more models.
arXiv (Cornell University)RACE: Large-scale ReAding Comprehension Dataset From Examinations
110 Citations2017Guokun Lai, Qizhe Xie +3 more
Who did What: A Large-Scale Person-Centered Cloze Dataset
105 Citations2016Takeshi Onishi, Hai Wang +3 more
A new "Who-did-What" dataset of over 200,000 fill-in-the-gap (cloze) multiple choice reading comprehension problems constructed from the LDC English Gigaword newswire corpus is constructed and proposed as a challenge task for the community.
arXiv (Cornell University)A BERT Baseline for the Natural Questions
97 Citations2019Chris Alberti, Kenton Lee +1 more
A new baseline for the Natural Questions is described and the gap between the model F1 scores reported in the original dataset paper and the human upper bound is reduced by 30% and 50% relative for the long and short answer tasks respectively.
Transactions of the Association for Computational LinguisticsCoQA: A Conversational Question Answering Challenge
97 Citations2019Siva Reddy, Danqi Chen +1 more
CoQA is introduced, a novel dataset for building Conversational Question Answering systems and it is shown that conversational questions have challenging phenomena not present in existing reading comprehension datasets (e.g., coreference and pragmatic reasoning).