The TUNA-REG Challenge 2009
Published 1 January 2009Open access
Albert Gatt, Anja Belz, Eric Kow
Citations95
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The GREC Task at REG '08 required participating systems to select coreference chains to the main subject of short encyclopaedic texts collected from Wikipedia, and the evaluation methods were described and described.
Abstract
The TUNA-REG'09 Challenge was one of the shared-task evaluation competitions at Generation Challenges 2009. TUNA-REG'09 used data from the TUNA Corpus of paired representations of entities and human-authored referring expressions. The shared task was to create systems that generate referring expressions for entities given representations of sets of entities and their properties. Four teams submitted six systems to TUNA-REG'09. We evaluated the six systems and two sets of human-authored referring expressions using several automatic intrinsic measures, a human-assessed intrinsic evaluation and a human task performance experiment. This report describes the TUNA-REG task and the evaluation methods used, and presents the evaluation results.
Keywords
Computer Science
Behavior Research Methods, Instruments, & ComputersDMDX: A Windows display program with millisecond accuracy
2,750 Citations2003Kenneth I. Forster, Jonathan C. Forster
DMDX is a Windows-based program designed primarily for language-processing experiments that uses the features of Pentium class CPUs and the library routines provided in DirectX to provide accurate timing and synchronization of visual and audio output.
Zeitschrift für Anglistik und AmerikanistikThe Cambridge Grammar of the English Language
2,417 Citations2005R. E. Huddleston, Geoffrey K. Pullum
Psychological BulletinMethods for dealing with reaction time outliers.
2,086 Citations1993Roger Ratcliff
It is concluded using quantitative examples that robust measures are much less affected by outliers and cutoffs than measures based on moments, and fitting explicit distribution functions as a way of recovering means and standard deviations is probably not worth routine use.
A model-theoretic coreference scoring scheme
682 Citations1995Marc Vilain, John D. Burger +3 more
This note describes a scoring scheme for the coreference task in MUC6 that improves on the original approach by grounding the scoring scheme in terms of a model; producing more intuitive recall and precision scores; and not requiring explicit computation of the transitive closure of coreference.
On coreference resolution performance metrics
535 Citations2005Xiaoqiang Luo
The paper proposes a Constrained Entity-Alignment F-Measure (CEAF) for evaluating coreference resolution and shows that the best alignment is a maximum bipartite matching problem which can be solved by the Kuhn-Munkres algorithm.
Information Processing & ManagementTwo uses of anaphora resolution in summarization
161 Citations2007Josef Steinberger, Massimo Poesio +2 more
A new method for using anaphoric information in Latent Semantic Analysis (LSA) is proposed, and its application to develop an LSA-based summarizer which achieves a significantly better performance than a system not using anphoric information.
Columbia Academic Commons (Columbia University)Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation
137 Citations2006Rebecca J. Passonneau
MASI, distance metric for comparing sets, is discussed and its use in quantifying the reliability of a specific dataset is illustrated, and it is argued that a paradigmatic reliability study should relate measures of inter-annotator agreement to independent assessments, such as significance tests of the annotated variables with respect to other phenomena.
Evaluation metrics for generation
127 Citations2000Srinivas Bangalore, Owen Rambow +1 more
It is confirmed that intrinsic metrics cannot replace human evaluation, but some correlate significantly with human judgments of quality and understandability and can be used for evaluation during development.
Evaluating algorithms for the generation of referring expressions using a balanced corpus
105 Citations2007Albert Gatt, Ielka van der Sluis +1 more
A corpus-based evaluation methodology, applied to a number of classic algorithms, that focuses on balance and semantic transparency to enable comparison of human and algorithmic output.
ArXiv.orgA Public Reference Implementation of the RAP Anaphora Resolution Algorithm
75 Citations2004Long Qiu, Min‐Yen Kan +1 more
This paper describes a standalone, publicly-available implementation of the Resolution of Anaphora Procedure (RAP), which resolves third person pronouns, lexical anaphors, and identifies pleonastic pronouns and provides a reference implementation that can be benchmarked against current algorithms.
OAR@UM (University of Malta)The Attribute Selection for GRE Challenge: Overview and Evaluation Results
50 Citations2007Anja Belz, Albert Gatt
) Challenge was the first shared-task evaluation challenge of Natural Language Generation and all submitted systems were tested automat-ically for minimality, uniqueness and ‘hu-manlikeness’.
Intrinsic vs. extrinsic evaluation measures for referring expression generation
48 Citations2008Anja Belz, Albert Gatt
It is found that there are no significant correlations between intrinsic and extrinsic evaluation measures for this task, and the kind of intrinsic evaluation metrics characteristic of current comparative HLT evaluation are applied.
Attribute selection for referring expression generation
25 Citations2008Albert Gatt, Anja Belz
This paper analyses the systems that participated in the first Shared Task Challenge in NLG in terms of their algorithmic properties, comparing new techniques to classic ones, based on results from a new human task-performance experiment and from the intrinsic measures that were used in the Challenge.
ScholarlyCommons (University of Pennsylvania)Entity-driven Rewrite for Multi-document Summarization
23 Citations2008Ani Nenkova
Sumaries produced using entity-driven rewrite have higher linguistic quality than a comparison non-extractive system and some improvement is also seen in content selection over extractive summarization as measured by pyramid method evaluation.
Generation of repeated references to discourse entities
19 Citations2007Anja Belz, Sebastian Varges
This paper addresses the complementary problem of generating repeated, potentially different referential expressions that refer to the same entity in the context of a piece of discourse longer than a sentence.
ScholarlyCommons (University of Pennsylvania)Using semantic relations to improve information retrieval
16 Citations2005Thomas E. Morton, Martha Palmer
This thesis examines passage retrieval for a question-answering task, and demonstrates that natural language techniques for determining categorical and referential relationships improve passage retrieval, and examines the integration of these relationships into two passage retrieval systems.