Methodology for Reliable Schema Development and Evaluation of Manual Annotations
Published 1 January 2003
Petra Saskia Bayerl, Ulrike Gut, Karsten I. Paul
Citations26
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A systematic approach to schema development is developed, which allows for developing coding schemas for fine-grained semantic annotations while systematically securing the quality of such annotations.
Abstract
The quality of manual annotations of linguistic data depends on the use of reliable coding schemas as well as on the abil-ity of human annotators to handle them appropriately. As is well known from a wide range of previous experiences an-notations using highly complex coding schemas often lead to unacceptable annotation quality. Reducing complexity might make schemas easier to handle, but in this way valuable in-formation needed for more sophisticated applications is ex-cluded as well. In order to deal with this problem, we devel-oped a systematic approach to schema development, which allows for developing coding schemas for fine-grained se-mantic annotations while systematically securing the quality of such annotations. For illustration, we present examples from two projects where text and speech data are annotated.
Keywords
Computer Science
BiometricsThe Measurement of Observer Agreement for Categorical Data
78,934 Citations1977J. Richard Landis, Gary G. Koch
A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
The Encyclopedia of Political ScienceReliability and Validity Assessment
5,079 Citations2011Edward G. Carmines
PsychometrikaNote on the Sampling Error of the Difference Between Correlated Proportions or Percentages
4,283 Citations1947Quinn McNemar
Two formulas are presented for judging the significance of the difference between correlated proportions and the chi square equivalent of one of the developed formulas.
Journal of Clinical EpidemiologyHigh agreement but low Kappa: I. the problems of two paradoxes
2,886 Citations1990Alvan R. Feinstein, Domenic V. Cicchetti
In a fourfold table showing binary agreement of two observers, the observed proportion of agreement, p0, can be paradoxically altered by the chance-corrected ratio that creates kappa as an index of concordance.
Journal of Clinical EpidemiologyBias, prevalence and kappa
2,106 Citations1993Ted Byrt, Janet Bishop +1 more
New indices that provide independent measures of bias and prevalence, as well as of observed agreement, are defined and a simple formula is derived that expresses kappa in terms of these three indices.
arXiv (Cornell University)Assessing agreement on classification tasks: the kappa statistic
2,092 Citations1996Jean Carletta
Educational and Psychological MeasurementCoefficient Kappa: Some Uses, Misuses, and Alternatives
1,544 Citations1981Robert L. Brennan, Dale J. Prediger
Medical Entomology and ZoologyHuman Reliability Analysis: Context and Control
738 Citations1994Erik Hollnagel
Performance, Reliability, And Unwanted Consequences, and the Need for Models of the Reliability of Cognition.
Development and use of a gold-standard data set for subjectivity classifications
498 Citations1999Janyce Wiebe, Rebecca Bruce +1 more
Bias-corrected tags are formulated and successfully used to guide a revision of the coding manual and develop an automatic classifier.
BiometrikaA TEST FOR HOMOGENEITY OF THE MARGINAL DISTRIBUTIONS IN A TWO-WAY CLASSIFICATION
462 Citations1955Alan Stuart
Computational LinguisticsThe reliability of a dialogue structure coding scheme
401 Citations1997Jean Carletta, Stephen Isard +4 more
This paper describes the reliability of a dialogue structure coding scheme based on utterance function, game structure, and higher-level transaction structure that has been applied to a corpus of spontaneous task-oriented spoken dialogues.
The British Journal of PsychiatryComparing the Classification of Subjects by Two Independent Judges
243 Citations1970A. E. Maxwell
Argumentative zoning information extraction from scientific text
225 Citations1999Simone Teufel
An annotation scheme for discourse-level argumentation in research articles
186 Citations1999Simone Teufel, Jean Carletta +1 more
This paper introduces an annotation scheme for scientific articles which can be used to build robust automatic abstracting systems in a consistent way and shows that the scheme is stable, reproducible and intuitive to use.
Natural Language EngineeringRecognizing subjectivity: a case study in manual tagging
174 Citations1999Rebecca Bruce, Janyce Wiebe
A case study of a sentence-level categorization in which tagging instructions are developed and used by four judges to classify clauses from the Wall Street Journal as either subjective or objective.
PsychometrikaSample Size Determinations for the Two Rater Kappa Statistic
157 Citations1988Virginia F. Flack, A. A. Afifi +2 more
Experiments in Constructing a Corpus of Discourse Trees
71 Citations1999Daniel Marcu, Estibaliz Amorrortu +1 more
The statistical measure used suggests that annotators can achieve good levels of agreement on the task of determining the high-level, rhetorical structure of texts and that building discourse parsers that incrementally derive correct rhetorical structures of unrestricted texts without applying any form of backtracking is feasible.
A Case Study on Inter-Annotator Agreement for Word Sense Disambiguation
69 Citations1999Hwee Tou Ng, Chung Yong Lim +1 more
This paper examines th~s msue by comparing the agreement rate on a large corpus of more than 30,000 sense-tagged instances of the WORDNET Semcor corpus and the DSO corpus, which has been independently tagged by two separate groups of human annotators.
Speech CommunicationAutomatic segmentation and labelling of multi-lingual speech data
38 Citations1996Annemie Vorstermans, J.-P. Martens +1 more
The results show that the accuracy of the obtained automatic segmentation and labelling is comparable to that of human experts.
Electronic workshops in computingText-level Structure of Research Papers : Implications for Text-Based Information Processing Systems
34 Citations1997Noriko Kando
It is shown that searching full-length texts using text-level structure achieved higher precision, compared to the searching without it, and suggested the application of text- level structure for text-based information systems, including passage extraction, browsing, and navigation within/across texts.
A bootstrapping approach to automating prosodic annotation for limited-domain synthesis
10 Citations2004Ivan Bulyko, Mari Ostendorf
A perceptual experiment shows that using predominantly automatic prosody labels the authors can achieve nearly as high synthesis quality as if all data was hand-labeled.
SciDok (Saarland University and State Library)Evaluating a scheme for dialogue annotation
4 Citations1997Elisabeth Maier
Using the kappa method for measuring the reliability of this coding scheme for the segmentation and annotation of transliterated spoken dialogues with dialogue act information, it is shown that this observation holds for an evaluation of intercoder reliability and for the stability of coding by one coder over time.