Evaluating message understanding systems: an analysis of the third message understanding conference (MUC-3)
Computational LinguisticsPublished 1 September 1993
Nancy Chinchor, David Lewis, Lynette Hirschman
Citations139
SJR quartileQ1
SJR score1.15
SNIP4.16
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The purpose, history, and methodology of the conference are reviewed, the participating systems are summarized, issues of measuring system effectiveness are discussed, the linguistic phenomena tests are described, and a critical look at the evaluation in terms of the lessons learned is provided.
Abstract
This paper describes and analyzes the results of the Third Message Understanding Conference (MUC-3). It reviews the purpose, history, and methodology of the conference, summarizes the participating systems, discusses issues of measuring system effectiveness, describes the linguistic phenomena tests, and provides a critical look at the evaluation in terms of the lessons learned. One of the common problems with evaluations is that the statistical significance of the results is unknown. In the discussion of system performance, the statistical significance of the evaluation results is reported and the use of approximate randomization to calculate the statistical significance of the results of MUC-3 is described.
Keywords
Computer Science
ScienceStatistical Data Analysis in the Computer Age
1,203 Citations1991Bradley Efron, Robert Tibshirani
Most of the authors' familiar statistical methods, such as hypothesis testing, linear regression, analysis of variance, and maximum likelihood estimation, were designed to be implemented on mechanical calculators.
Journal of the Operational Research SocietyComputer Intensive Methods for Testing Hypotheses: An Introduction
667 Citations1990K. J. Evans, Eric W. Noreen
Procedure for quantitatively comparing the syntactic coverage of English grammars
436 Citations1991Judith L. Klavans, Mark Liberman +12 more
ScholarlyCommons (University of Pennsylvania)Part-of-Speech Tagging Guidelines for the Penn Treebank Project (3rd Revision)
436 Citations1990Beatrice Santorini
This manual addresses the linguistic issues that arise in connection with annotating texts by part of speech ("tagging") and discusses parts of speech that are easily confused and gives guidelines on how to tag such cases.
American DocumentationEffectiveness of information retrieval methods
152 Citations1969John A. Swets
Results of some 50 different retrieval methods applied in three experimental retrieval systems were subjected to the analysis suggested by statistical decision theory and validates a previously‐proposed measure of effectiveness and demonstrates its several desirable properties.
Journal of the Operational Research SocietyProblem Solving: A Statistician's Guide.
141 Citations1996Stelios Kafandaris, Chris Chatfield
The statistical significance of the MUC-5 results
58 Citations1993Nancy Chinchor
The MUC-4 scores of recall, precision, and the F-measures are used to measure the performance of the participating systems and the method of hypothesis testing used is a computationally-intensive method known as approximate randomization.
Workshop on the evaluation of natural language processing systems
42 Citations1990Martha Palmer, Tim Finin
In the past few years, the computational linguistics research community has begun to wrestle with the problem of how to evaluate its progress in developing natural language processing systems, and there has been little agreement about training sets and test sets, or about clearly defined subsets of problems that constitute standards for levels of performance.
Subject-based evaluation measures for interactive spoken language systems
29 Citations1992Patti Price, Lynette Hirschman +2 more
This paper proposes a within-subject design comparing the results of a software-sharing exercise carried out jointly by MIT and SRI, and suggests either a large number of subjects or aWithin- subject design is necessary.
Comparing MUCK-II and MUC-3
12 Citations1991Lynette Hirschman
The natural language community has made impressive progress in evaluation over the last four years, but as the evaluations become more sophisticated and more ambitious, a fundamental problem emerges: how to compare results across changing evaluation paradigms.
An adjunct test for discourse processing in MUC-4
11 Citations1992Lynette Hirschman
The MUC-4 adjunct test is defined to examine two hypotheses related to discourse complexity and expected system performance: The more complex the distribution of the source information for filling a given slot or template, the more difficult it is to process the message correctly.
Data extraction as text categorization
10 Citations1991David Lewis
The data extraction systems studied in the MUC-3 evaluation perform a variety of subtasks in filling out templates that seem to require a system to represent the structure of a text in some detail to perform the task successfully.
MUC-3 linguistic phenomena test experiment
9 Citations1991Nancy Chinchor
The experiment performed as part of the MUC-3 evaluation was aimed at determining whether linguistic phenomena can be tested in isolation for state-of-the-art data extraction, and the results indicate that linguistic phenomena are isolatable and that performance for linguistic phenomenaCan be measured using the M UC-3 scoring system.
GE adjunct test report
3 Citations1992George Krupka, Lisa F. Rau
The results indicate that the current MUC-4 "flat" templade design with cross-references closely approximates a true object-oriented design, however the object- oriented design allows for additional performance data to be calculated, facilitating diagnosis.