Automatic Evaluation of Information Ordering: Kendall's Tau
Computational LinguisticsPublished 21 November 2006Open access
Mirella Lapata
Citations194
SJR quartileQ1
SJR score1.15
SNIP4.16
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An evaluation method based on Kendall's, a metric of rank correlation, is proposed that is inexpensive, robust, and representation independent and it is shown that Kendall's correlates reliably with human ratings and reading times.
Abstract
This article considers the automatic evaluation of information ordering, a task underlying many text-based applications such as concept-to-text generation and multidocument summarization. We propose an evaluation method based on Kendall's τ, a metric of rank correlation. The method is inexpensive, robust, and representation independent. We show that Kendall's τ correlates reliably with human ratings and reading times.
Keywords
Computer Science
Minimum error rate training in statistical machine translation
2,766 Citations2003Franz Josef Och
It is shown that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.
Journal of Artificial Intelligence ResearchSemantic Similarity in a Taxonomy: An Information-Based Measure and its Application to Problems of Ambiguity in Natural Language
2,085 Citations1999Philip Resnik
This article presents a measure of semantic similarity in an IS-A taxonomy based on the notion of shared information content that performs better than the traditional edge-counting approach.
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics
1,605 Citations2002George R. Doddington
NIST commissioned NIST to develop an MT evaluation facility based on the IBM work, which is now available from NIST and serves as the primary evaluation measure for TIDES MT research.
Automatic evaluation of summaries using N-gram co-occurrence statistics
1,588 Citations2003Chin-Yew Lin, Eduard Hovy
The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.
Psychological ReviewInference during reading.
1,039 Citations1992Gail McKoon, Roger Ratcliff
The minimalist hypothesis is proposed, and it is argued that automatically encoded minimalist inferences provide the basic representation of textual information from which more goal-directed, purposeful inferences are constructed.
Computer systems that learn: classification and prediction methods from statistics, neural nets, machine learning, and expert systems
906 Citations1991Sholom M. Weiss, Casimir A. Kulikowski
This book discusses how to Estimate the True Performance of a Learning System, and the Importance of Unbiased Error Rate Estimation, and Machine Learning: Easily Understood Decision Rules.
Natural language processingAutomatic Summarization
839 Citations2001Inderjeet Mani
The challenges that remain open, in particular the need for language generation and deeper semantic understanding of language that would be necessary for future advances in the field are discussed.
Wiley Encyclopedia of Electrical and Electronics EngineeringComputational Linguistics
834 Citations1999Fred Popowich
Journal of Experimental Psychology Learning Memory and CognitionRegression analyses of repeated measures data in cognitive research.
712 Citations1990Robert F. Lorch, Jerome L. Myers
Re-evaluating the Role of Bleu in Machine Translation Research
621 Citations2006Chris Callison-Burch, Miles Osborne +1 more
It is shown that an improved Bleu score is neither necessary nor sufficient for achieving an actual improvement in translation quality, and two significant counterexamples to Bleu’s correlation with human judgments of quality are given.
Generating natural language summaries from multiple on-line sources
381 Citations1998Dragomir Radev, Kathleen McKeown
Journal of Artificial Intelligence ResearchInferring Strategies for Sentence Ordering in Multidocument News Summarization
333 Citations2002Regina Barzilay, Noémie Elhadad
A strategy for ordering information that combines constraints from chronological order of events and topical relatedness is implemented and Evaluation of the augmented algorithm shows a significant improvement of the ordering over two baseline strategies.
Journal of Educational PsychologyUsing Kintsch's computational model to improve instructional text: Effects of repairing inference calls on recall and cognitive structures.
316 Citations1991Bruce K. Britton, Sami̇ Gülgöz
Kintsch's reading comprehension model was used to identify locations where inferences were called for in a 1000-word expository text and each location was repaired to produce a principled revision.
arXiv (Cornell University)Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization
299 Citations2004Regina Barzilay, Lillian Lee
An effective knowledge-lean method for learning content models from unannotated documents is presented, utilizing a novel adaptation of algorithms for Hidden Markov Models and applied to two complementary tasks: information ordering and extractive summarization.
Journal of Psychiatric ResearchCorrelating and predicting psychiatric symptom ratings: Spearmans r versus Kendalls tau correlation
277 Citations1999Stephan Arndt, Carolyn Turvey +1 more
Kendall's tau(b) has many advantages over Pearson's and Spearman's r; when applied to psychiatric data, tau (b) maintained adequate control of type I errors, was nearly as powerful as Pearson's r, provided much tighter confidence intervals and had a clear interpretation.
Probabilistic text structuring
265 Citations2003Mirella Lapata
An approach to information ordering that is particularly suited for text-to-text generation and the appropriateness of such a model for multidocument summarization is assessed.
Cranking: Combining Rankings Using Conditional Probability Models on Permutations
179 Citations2002Guy Lebanon, John Lafferty
A new approach to ensemble learning is introduced that takes ranking rather than classification as fundamental, leading to models on the symmetric group and its cosets, a generalization of the Mallows model on permutations to combine multiple input rankings.
Columbia Academic Commons (Columbia University)Automatic Text Summarization of Newswire: Lessons Learned from the Document Understanding Conference
156 Citations2005Ani Nenkova
An overview of the achieved results in the different types of summarization tasks, comparing both the broader classes of baselines, systems and humans, as well as individual pairs of summarizers (both human and automatic).
Evaluation metrics for generation
127 Citations2000Srinivas Bangalore, Owen Rambow +1 more
It is confirmed that intrinsic metrics cannot replace human evaluation, but some correlate significantly with human judgments of quality and understandability and can be used for evaluation during development.
Correlating automated and human assessments of machine translation quality
117 Citations2003Deborah A. Coughlin
It is suggested that when human evaluators are forced to make decisions without sufficient context or domain expertise, they fall back on strategies that are not unlike determining n-gram precision.
Statistics in Psycholinguistics: A critique of some current gold standards
111 Citations2004R. Harald Baayen
A detailed critique of some current gold standards for the statistical analysis of experimental data in psycholinguistics is presented, highlighting the disadvantages of reducing numerical variables to factors and the importance of including available covariates in the model.
Information fusion for multidocument summarization: paraphrasing and generation
93 Citations2003Kathleen McKeown, Regina Barzilay
This thesis presents unsupervised learning techniques to identify paraphrasing given a corpus of multiple parallel texts, and shows that the algorithm extracts paraphrases with high accuracy and significantly outperforms a state of the art algorithm developed for related tasks in machine translation.
Computational LinguisticsA Probabilistic Account of Logical Metonymy
91 Citations2003Maria Lapata, Alex Lascarides
This article acquires the meanings of metonymic verbs and adjectives from a large corpus and proposes a probabilistic model that provides a ranking on the set of possible interpretations and identifies the interpretations automatically by exploiting the consistent correspondences between surface syntactic cues and meaning.
Edinburgh Research Explorer (University of Edinburgh)Experiments Using Stochastic Search for Text Planning
75 Citations1998Chris Mellish, Alistair Knott +2 more
This work takes as its starting point Marcu's excellent formalisation of R S T and the problem of building legal RST trees, and for the purposes of this paper the phrase "text planning" will generally denote the task characterised by him.
Digital Repository at the University of Maryland (University of Maryland College Park)Measuring Verb Similarity
64 Citations2000Philip Resnik, Mona Diab
Several models of semantic similarity are presented, based on differing rep- resentational assumptions, and their properties are investigated via comparison with human ratings of verb similarity to offer insight into the bases for human similarity judgments.
Evaluating Information Content by Factoid Analysis : Human annotation and stability
58 Citations2004Stephanie Teufel, Hans van Halteren
It is shown that factoid annotation is highly re-producible, a weighted factoid score is introduced, how many summaries are required for stable system rankings is estimated, and it is shown that the factoid scores cannot be su–ciently approximated by unigrams and the DUC information overlap measure.
Empirically estimating order constraints for content planning in generation
51 Citations2001Pablo Duboue, Kathleen McKeown
A system to automatically learn elements of a plan and the ordering constraints among them and a novel algorithm to find parallels between transcripts based on techniques used in computational genomics is developed.
Improving chronological sentence ordering by precedence relation
50 Citations2004Naoaki Okazaki, Yutaka Matsuo +1 more
An approach to coherent sentence ordering for summarizing newspaper articles by combining the refinement algorithm with topical segmentation and chronological ordering and results reveal that the proposed method improves chronological sentence ordering.
Stochastic Text Structuring Using the Principle of Continuity
41 Citations2002Nikiforos Karamanis, Hisar Maruli Manurung
By employing an elitist strategy for stochastic search it is possible to quickly reach the global optimum of minimal violations of continuity, chosen over other more complicated metrics of text coherence.
Columbia Academic Commons (Columbia University)Content planner construction via evolutionary algorithms and a corpus-based fitness function
38 Citations2002Pablo Duboue, Kathleen McKeown
This paper presents a novel technique to learn a tree-like structure for a content planner from an aligned corpus of semantic inputs and corresponding, human-produced, outputs by applying a stochastic search mechanism with a two-levelness function.
Information SciencesA preference learning approach to sentence ordering for multi-document summarization
37 Citations2012Danushka Bollegala, Naoaki Okazaki +1 more
The proposed sentence ordering algorithm considers pairwise comparisons between sentences to determine a total ordering, using a greedy search algorithm, thereby avoiding the combinatorial time complexity typically associated with total ordering tasks.
Edinburgh Research Archive (University of Edinburgh)Entity Coherence for Descriptive Text Structuring
35 Citations2004Nikiforos Karamanis
This thesis provides substantial insight into the role of entity coherence as a descriptive textStructuring constraint and introduces a general corpus-based methodology to identify which of the many formulations of Centering best suits text structuring.
arXiv (Cornell University)Learning to Order Facts for Discourse Planning in Natural Language Generation
17 Citations2003Aggeliki Dimitromanolaki, Ion Androutsopoulos
A quantitative evaluation performed in the domain of museum exhibit descriptions indicates that the machine learning approach to discourse planning in natural language generation performs significantly better than manually constructed ordering rules.
Journal of Logic Language and InformationSupplementing Entity Coherence with Local Rhetorical Relations for Information Ordering
11 Citations2007Nikiforos Karamanis
The results indicate that the simplest metric continues to perform better than its competitors even when local rhetorical coherence is taken into account, and this extra coherence constraint decreases its performance.
Using a Corpus of Sentence Orderings Defined by Many Experts to Evaluate Metrics of Coherence for Text Structuring
9 Citations2005Nikiforos Karamanis, Chris Mellish
This paper addresses two previously unresolved issues in the automatic evaluation of Text Structuring in Natural Language Generation (NLG) by describing how to verify the generality of an existing collection of sentence orderings using data provided by additional experts.