Message Understanding Conference (MUC) tests of discourse processing

Published 1 January 1995

Nancy Chinchor, Beth Sundheim

Citations20

TL;DR

An early attempt to use the results from an information extraction evaluation to provide insight notice relationship between the difficulty of discourse processing and performance on the information extraction task and an upcoming noun phrase coreference evaluation is described.

Abstract

sundheim @ nose.rail Performance evaluations of NLP systems have been designed and conducted that require systems to extract cer-tain prespecified information about events and entities. A single text may describe multiple events and entities, and the evaluation task requires the system to resolve references to produce the expected output.We describe an early attempt to use the results from an information extraction evaluation to provide insight notice relationship between the difficulty of discourse processing and performance on the information extraction task. We then discuss an upcoming noun phrase coreference evaluation that has been designed indepen-dently of any other evaluation task in order to establish a clear performance benchmark on a small set of discourse phenomena. Background on the MUC Evaluations Five Message Understanding Conferences have been held since 1987 (Sundheim and Chinchor 1993) and a sixth one

Keywords

Arts and HumanitiesHealth Professions

A model-theoretic coreference scoring scheme

682 Citations1995Marc Vilain, John D. Burger +3 more

This note describes a scoring scheme for the coreference task in MUC6 that improves on the original approach by grounding the scoring scheme in terms of a model; producing more intuitive recall and precision scores; and not requiring explicit computation of the transitive closure of coreference.

View PDF

Computational LinguisticsEvaluating message understanding systems: an analysis of the third message understanding conference (MUC-3)

139 Citations1993Nancy Chinchor, David Lewis +1 more

The purpose, history, and methodology of the conference are reviewed, the participating systems are summarized, issues of measuring system effectiveness are discussed, the linguistic phenomena tests are described, and a critical look at the evaluation in terms of the lessons learned is provided.

Overview of the fourth message understanding evaluation and conference

133 Citations1992Beth Sundheim

The Fourth Message Understanding Conference (MUC-4) is the latest in a series of conferences that concern the evaluation of natural language processing (NLP) systems and reports on progress made both in the development of systems capable of analyzing relatively short English texts and in the definition of a rigorous performance evaluation methodology.

View PDF

Overview of the third message understanding evaluation and conference

65 Citations1991Beth Sundheim

The Naval Ocean Systems Center (NOSC) has conducted the third in a series of evaluations of English text analysis systems, intended to advance understanding of the merits of current text analysis techniques, as applied to the performance of a realistic information extraction task.

View PDF

Survey of the Message Understanding Conferences

29 Citations1993Beth Sundheim, Nancy Chinchor

The role of the conferences in the evaluation of information extraction systems is assessed in terms of the purposes of three broad classes of evaluation: progress, adequacy, and diagnostic.

A language-independent anaphora resolution system for understanding multilingual texts

23 Citations1993Chinatsu Aone, Douglas McKee

A new discourse module within the multilingual NLP system that is language-independent, robust and trainable using discourse-tagged corpora, and separating discourse phenomena from knowledge sources makes the discourse module easily extensible to additional phenomena.

View PDF

Whither written language evaluation?

20 Citations1994Ralph Grishman

Common evaluations have grown to be a major component of all the ARPA Human Language Technology programs and have been a major impetus in the development of systems for performing such "information extraction" tasks, and thus in demonstrating the potential practical value of some of the written language processing technology.

View PDF

Comparing human and machine performance for natural language information extraction

17 Citations1993Craig A. Will

Results are presented from a comparative study of human and machine performance for one of the information extraction tasks used in the MUC-5/Tipster evaluation that can help assess the maturity and applicability of the technology.

View PDF

Computational aspects of discourse in the context of MUC-3

16 Citations1991Lucja Iwańska, Douglas E. Appelt +7 more

This paper gives an overview of the modules that make the transition from the interpretations of sentences to the interpretation of the text that contains these sentences and exhibits various degrees of the following discourse understanding capabilities.

View PDF

An adjunct test for discourse processing in MUC-4

11 Citations1992Lynette Hirschman

The MUC-4 adjunct test is defined to examine two hypotheses related to discourse complexity and expected system performance: The more complex the distribution of the source information for filling a given slot or template, the more difficult it is to process the message correctly.

View PDF