Survey of the Message Understanding Conferences
Published 1 January 1993
Beth Sundheim, Nancy Chinchor
Citations29
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The role of the conferences in the evaluation of information extraction systems is assessed in terms of the purposes of three broad classes of evaluation: progress, adequacy, and diagnostic.
Abstract
In this paper, the Message Understanding Conferences are reviewed, and the natural language system evaluation that is underway in preparation for the next conference is described. The role of the conferences in the evaluation of information extraction systems is assessed in terms of the purposes of three broad classes of evaluation: progress, adequacy, and diagnostic. The conferences have measured system performance primarily to assess progress and the state of the art, but they have also been influenced by the concerns associated with assessing adequacy and providing diagnostics. Challenges for the future of similar evaluations are also discussed.
Keywords
Computer Science
MUC-4 evaluation metrics
548 Citations1992Nancy Chinchor
The scoring algorithms used to arrive at the metrics as well as the improvements that were made to the MUC-3 methods were described, showing that the M UC-4 systems' scores represent a larger improvement over MUC -3 performance than the numbers themselves suggest.
Computational LinguisticsEvaluating message understanding systems: an analysis of the third message understanding conference (MUC-3)
139 Citations1993Nancy Chinchor, David Lewis +1 more
The purpose, history, and methodology of the conference are reviewed, the participating systems are summarized, issues of measuring system effectiveness are discussed, the linguistic phenomena tests are described, and a critical look at the evaluation in terms of the lessons learned is provided.
Overview of the fourth message understanding evaluation and conference
133 Citations1992Beth Sundheim
The Fourth Message Understanding Conference (MUC-4) is the latest in a series of conferences that concern the evaluation of natural language processing (NLP) systems and reports on progress made both in the development of systems capable of analyzing relatively short English texts and in the definition of a rigorous performance evaluation methodology.
MUC-3 evaluation metrics
105 Citations1991Nancy Chinchor
The MUC-3 evaluation metrics are measures of performance for the M UC-3 template fill task that were adapted from traditional measures in information retrieval and signal procesing and were still evolving to fit the more complex data extraction task of Muc-3 when the evaluation was performed.
The statistical significance of the MUC-5 results
58 Citations1993Nancy Chinchor
The MUC-4 scores of recall, precision, and the F-measures are used to measure the performance of the participating systems and the method of hypothesis testing used is a computationally-intensive method known as approximate randomization.
The statistical significance of the MUC-4 results
56 Citations1992Nancy Chinchor
The MUC-4 scores of recall, precision, and the F-measures are used to measure the performance of the participating systems and the method of hypothesis testing used is a computationally-intensive method known as approximate randomization.
Third message understanding evaluation and conference (MUC-3)
19 Citations1991Beth Sundheim
The Naval Ocean Systems Center is conducting the third in a series of evaluations of English text analysis systems, finding that task-oriented tests enable straightforward comparisons among systems and provide useful quantitative data on the state of the art in text understanding.
Plans for a task-oriented evaluation of natural language understanding systems
17 Citations1989Beth Sundheim
A plan is presented for evaluating natural language processing systems that have focused on the issues of text understanding as exemplified in short texts from military messages, and definition of a simulated database update task that requires NLP systems to fill a template with information found in the texts.
Computational aspects of discourse in the context of MUC-3
16 Citations1991Lucja Iwańska, Douglas E. Appelt +7 more
This paper gives an overview of the modules that make the transition from the interpretations of sentences to the interpretation of the text that contains these sentences and exhibits various degrees of the following discourse understanding capabilities.
Comparing MUCK-II and MUC-3
12 Citations1991Lynette Hirschman
The natural language community has made impressive progress in evaluation over the last four years, but as the evaluations become more sophisticated and more ambitious, a fundamental problem emerges: how to compare results across changing evaluation paradigms.
An adjunct test for discourse processing in MUC-4
11 Citations1992Lynette Hirschman
The MUC-4 adjunct test is defined to examine two hypotheses related to discourse complexity and expected system performance: The more complex the distribution of the source information for filling a given slot or template, the more difficult it is to process the message correctly.
MUC-3 linguistic phenomena test experiment
9 Citations1991Nancy Chinchor
The experiment performed as part of the MUC-3 evaluation was aimed at determining whether linguistic phenomena can be tested in isolation for state-of-the-art data extraction, and the results indicate that linguistic phenomena are isolatable and that performance for linguistic phenomenaCan be measured using the M UC-3 scoring system.