The TREC-8 Question Answering Track Report

Abstract

The TREC-8 Question Answering track was the rst large-scale evaluation of domain-independent question answering systems. This paper summarizes the results of the track by giving a brief overview of the di erent approaches taken to solve the problem. The most accurate systems found a correct response for more than 2/3 of the questions. Relatively simple bag-of-words approaches were adequate for nding answers when responses could be as long as a paragraph (250 bytes), but more sophisticated processing was necessary for more direct responses (50 bytes). The TREC-8 Question Answering track was an initial e ort to bring the bene ts of large-scale evaluation to bear on a question answering (QA) task. The goal in the QA task is to retrieve small snippets of text that contain the actual answer to a question rather than the document lists traditionally returned by text retrieval systems. The assumption is that users would usually prefer to be given the answer rather than nd the answer themselves in a document. This paper summarizes the retrieval results of the track; a companion paper (\\The TREC-8 Question Answering Track Evaluation") gives details about how the evaluation was implemented. By necessity, a track report can give only an overview of the di erent approaches used in the track. Readers are urged to