A Methodology for Extrinsic Evaluation of Text Summarization: Does ROUGE Correlate?
Published 1 June 2005
Bonnie J. Dorr, Christof Monz, Stacy President, Richard Schwartz, David Zajic
Citations37
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper demonstrates the usefulness of summaries in an extrinsic task of relevance judgment based on a new method for measuring agreement, Relevance-Prediction, which compares subjects’ judgments on summaries with their own judgments on full text documents.
Abstract
This paper demonstrates the usefulness of summaries in an extrinsic task of relevance judgment based on a new method for measuring agreement, Relevance-Prediction, which compares subjects’ judgments on summaries with their own judgments on full text documents. We demonstrate that this is a more reliable measure than previous measures based on gold standards. Because it it more reliable, we are able to make stronger statistical statements about the benefits of summarization. We were able to find positive correlations between ROUGE scores and two different summary types, where only weak or negative correlations were found using other agreement measures. We discuss the importance of this result and the implications for automatic evaluation of summarization in the future. 1
Keywords
Computer Science
Revue économiqueNonparametric Statistics for the Behavioral Sciences
35,492 Citations1958Henri Guitton, Sidney Siegel
ROUGE: A Package for Automatic Evaluation of Summaries
8,286 Citations2004Chin-Yew Lin
Four different RouGE measures are introduced: ROUGE-N, ROUge-L, R OUGE-W, and ROUAGE-S included in the Rouge summarization evaluation package and their evaluations.
arXiv (Cornell University)Assessing agreement on classification tasks: the kappa statistic
2,092 Citations1996Jean Carletta
Automatic evaluation of summaries using N-gram co-occurrence statistics
1,588 Citations2003Chin-Yew Lin, Eduard Hovy
The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.
Columbia Academic Commons (Columbia University)Evaluating Content Selection in Summarization: The Pyramid Method
537 Citations2004Ani Nenkova, Rebecca J. Passonneau
It is argued that the method presented is reliable, predictive and diagnostic, thus improves considerably over the shortcomings of the human evaluation method currently used in the Document Understanding Conference.
Computational LinguisticsThe Kappa Statistic: A Second Look
498 Citations2004Barbara Di Eugenio, Michael Glass
This squib discusses the assumptions underlying different computations of the expected agreement component of the kappa coefficient of agreement and discusses how prevalence and bias affect the measure.
Advantages of query biased summaries in information retrieval
438 Citations1998Anastasios Tombros, Mark Sanderson
Investigation into the utility of document summarisation in the context of information retrieval and the application of so called query biased (or user directed) summaries indicate that the use of query biased summaries significantly improves both the accuracy and speed of user relevance judgements.
Generating natural language summaries from multiple on-line sources
381 Citations1998Dragomir Radev, Kathleen McKeown
Medical Entomology and ZoologyEvaluating Natural Language Processing Systems: An Analysis and Review
276 Citations1996Karen Spärck Jones, Julia Galliers
Columbia Academic Commons (Columbia University)Summarization evaluation methods: Experiments and analysis
179 Citations1998Kathleen McKeown, Hongyan Jing +2 more
The results show that different parameters of an experiment can affect how well a system scores, and describe how parameters can be controlled to produce a sound evaluation.
Summarization Evaluation: An Overview.
156 Citations2001Inderjeet Mani
Natural Language EngineeringSUMMAC: a text summarization evaluation
144 Citations2002Inderjeet Mani, Gary Klein +4 more
Analysis of feedback forms filled in after each decision indicated that the intelligibility of present-day machine-generated summaries is high, and the evaluation methods used in the SUMMAC evaluation are of interest to both summarization evaluation as well as evaluation of other ‘output-related’ NLP technologies, where there may be many potentially acceptable outputs.
Examining the consensus between human summaries
90 Citations2003Hans van Halteren, Simone Teufel
This work presents a new approach to summary evaluation which combines two novel aspects, namely (a) content comparison between gold standard summary and system summary via factoids, a pseudo-semantic representation based on atomic information units which can be robustly marked in text, and (b) use of a gold standard consensus summary.
Columbia Academic Commons (Columbia University)Generating Natural Language Summaries from Multiple On-Line Sources: Language Reuse and Regeneration
19 Citations1999Kathleen McKeown
Evaluation of phrase-representation summarization based on information retrieval task
9 Citations2000Mamiko Oka, Y. Ueda
An improved task-based evaluation method of summarization is developed, the accuracy of which is increased by specifying the details of the task including background stories, and by assigning ten subjects per summary sample.
Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)Extrinsic Evaluation of Automatic Metrics for Summarization
9 Citations2004Bonnie J. Dorr, Christof Monz +4 more
It is shown that it is possible to save time using summaries for relevance assessment without adversely impacting the degree of accuracy that would be possible with full documents, and a small yet statistically significant correlation between some of the intrinsic measures and a user's performance in an extrinsic task is found.
Generic summaries for indexing in information retrieval – Detailed test results
2 Citations2021Tetsuya Sakai, Karen Spärck Jones