Annotating expressions of Appraisal in English
Language Resources and EvaluationPublished 11 December 2010Open access
Jonathon Read, John B. Carroll
Citations126
SJR quartileQ1
SJR score0.48
SNIP1.79
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Although the annotation task is difficult, there are many instances where the annotators agree; these are used to create a gold-standard corpus for future experimentation with Appraisal.
Abstract
In the context of Systemic Functional Linguistics, Appraisal is a theory describing the types of language utilised in communicating emotion and opinion. Robust automatic analyses of Appraisal could contribute in a number of ways to computational sentiment analysis by: distinguishing various types of evaluation, for example affect, ethics or aesthetics; discriminating between an author’s opinions and the opinions of authors referenced by the author and determining the strength of evaluations. This paper reviews the typology described by Appraisal, presents a methodology for annotating Appraisal, and the use of this to annotate a corpus of book reviews. It discusses an inter-annotator agreement study, and considers instances of systematic disagreement that indicate areas in which Appraisal may be refined or clarified. Although the annotation task is difficult, there are many instances where the annotators agree; these are used to create a gold-standard corpus for future experimentation with Appraisal.
Keywords
PsychologyComputer Science
BiometricsThe Measurement of Observer Agreement for Categorical Data
78,934 Citations1977J. Richard Landis, Gary G. Koch
A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
Language<b>Introduction to functional grammar</b> . By M. A. K. Halliday London & Baltimore: Edward Arnold, 1985. Pp. xxxv, 387.
9,924 Citations1989Christian M.I.M. Matthiessen
Thumbs up?
6,987 Citations2002Bo Pang, Lillian Lee +1 more
This work considers the problem of classifying documents not by topic, but by overall sentiment, e.g., determining whether a review is positive or negative, and concludes by examining factors that make the sentiment classification problem more challenging.
College Composition and CommunicationA Comprehensive Grammar of the English Language
5,535 Citations1987P Beauvais, Randolph Quirk +3 more
Meeting of the Association for Computational LinguisticsThumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews
3,654 Citations2002Peter Peter, Turney
The Language of Evaluation: Appraisal in English
2,779 Citations2005James R. Martin, Peter Robert Rupert White
arXiv (Cornell University)Thumbs up? Sentiment Classification using Machine Learning Techniques
2,208 Citations2002Bo Pang, Lillian Lee +1 more
arXiv (Cornell University)Assessing agreement on classification tasks: the kappa statistic
2,092 Citations1996Jean Carletta
TESOL QuarterlyDisciplinary Discourses: Social Interactions in Academic Writing
1,985 Citations2001Sun Jian-dong, Ken Hyland
Appraisal processes in emotion: Theory, methods, research.
1,911 Citations2001Klaus R. Scherer, Angela Schorr +1 more
On the algorithmic implementation of multiclass kernel-based vector machines
1,790 Citations2002Koby Crammer, Yoram Singer
This paper describes the algorithmic implementation of multiclass kernel-based vector machines using a generalized notion of the margin to multiclass problems, and describes an efficient fixed-point algorithm for solving the reduced optimization problems and proves its convergence.
Computers and the HumanitiesAnnotating Expressions of Opinions and Emotions in Language
1,770 Citations2005Janyce Wiebe, Theresa Wilson +1 more
The manual annotation process and the results of an inter-annotator agreement study on a 10,000-sentence corpus of articles drawn from the world press are presented.
ArXiv.orgThumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews
1,585 Citations2002Peter D. Turney
Language<b>Hedging in scientific research articles.</b> By Ken Hyland. (Pragmatics and beyond 54.) Amsterdam: John Benjamins, 1998. Pp. x, 307.
807 Citations1999Ingrid Piller
Computational LinguisticsLearning Subjective Language
743 Citations2004Janyce Wiebe, Theresa Wilson +3 more
This article shows that the density of subjectivity clues in the surrounding context strongly affects how likely it is that a word is subjective, and it provides the results of an annotation study assessing the subjectivity of sentences with high-density features.
Cognitive Systems ResearchA domain-independent framework for modeling emotion
725 Citations2004Jonathan Gratch, Stacy Marsella
It is shown how psychological theories of emotion shed light on the interaction between emotion and cognition, and thus can inform the design of human-like autonomous agents that must convey these core aspects of human behavior.
SemEval-2007 task 14
599 Citations2007Carlo Strapparava, Rada Mihalcea
The data set used in the evaluation and the results obtained by the participating systems are described, meant as an exploration of the connection between emotions and lexical semantics.
English for Specific PurposesDisciplinary discourses: social interactions in academic writing
569 Citations2001John M. Swales
Using appraisal groups for sentiment analysis
562 Citations2005Casey Whitelaw, Navendu Garg +1 more
A new method for sentiment classification based on extracting and analyzing appraisal groups such as ``very good'' or ``not terribly funny'' is presented, based on several task-independent semantic taxonomies based on Appraisal Theory.
The information retrieval seriesContextual Valence Shifters
518 Citations2006Livia Polanyi, Annie Zaenen
It is argued that the current work in this area that concentrates mainly on the negative or positive attitude communicated by individual terms is incomplete and often gives the wrong results when implemented directly and proposes a simple, “proof of concept” implementation for some contextual shifters.
Facial Expression Of Emotion
510 Citations2002Dacher Keltner, Paul Ekman +2 more
Determining the semantic orientation of terms through gloss classification
388 Citations2005Andrea Esuli, Fabrizio Sebastiani
This paper presents a new method for determining the orientation of subjective terms based on the quantitative analysis of the glosses of such terms given in on-line dictionaries, and on the use of the resulting term representations for semi-supervised term classification.
Meeting of the Association for Computational LinguisticsUsing Emoticons to reduce Dependency in Machine Learning Techniques for Sentiment Classification
375 Citations2005Jonathon, Read
Using emoticons to reduce dependency in machine learning techniques for sentiment classification
367 Citations2005Jonathon Read
This paper demonstrates that match with respect to domain and time is also important, and presents preliminary experiments with training data labeled with emoticons, which has the potential of being independent of domain, topic and time.
IEEE Transactions on Fuzzy SystemsAffect analysis of text using fuzzy semantic typing
291 Citations2001Pero Subašić, Alison K. Huettner
Seoul National University Open Repository (Seoul National University)100 Million Words of English:The British National Corpus (BNC)
198 Citations1992Geoffrey Leech
100 Million Words of English: The British National Corpus (BNC) is a celebration of 100 million words of English.
Analyzing Appraisal Automatically
180 Citations2004Maite Taboada, Jack Grieve
A standard method for calculating semantic orientation is applied, and a linguistic classification of Appraisal is applied that could be helpful in distinguishing different types of subjective texts (e.g., movie reviews from consumer product reviews).
Natural Language EngineeringRecognizing subjectivity: a case study in manual tagging
174 Citations1999Rebecca Bruce, Janyce Wiebe
A case study of a sentence-level categorization in which tagging instructions are developed and used by four judges to classify clauses from the Wall Street Journal as either subjective or objective.
Building emotion lexicon from weblog corpora
164 Citations2007Changhua Yang, Kevin Lin +1 more
A collocation model is proposed to learn emotion lexicons from weblog articles and emotion classification at sentence level is experimented by using the mined lexicons to demonstrate their usefulness.
Recognizing and Organizing Opinions Expressed in the World Press
74 Citations2003Janyce Wiebe, Eric Breck +10 more
The activities of the MPQA project were organized around an end-user task designed to utilize information about perspective—the task of clustering responses to yes/no questions based on perspective, and the results demonstrate the feasibility of annotatingInformation about perspective.
Lecture notes in computer scienceAutomatically Determining Attitude Type and Force for Sentiment Analysis
68 Citations2009Shlomo Argamon, K R Bloom +2 more
This chapter describes a method for the automatic determination of complex sentiment-related attributes such as attitude type and force by applying supervised learning to WordNet glosses, and shows that the method achieves good effectiveness, and is therefore well-suited to contexts in which these lexicons need to be generated from scratch.
Oxford University Research Archive (ORA) (University of Oxford)Sentiment Composition
67 Citations2007Karo Moilanen, Stephen Pulman
Natural Language EngineeringA definition and short history of Language Engineering
45 Citations1999Hamish Cunningham
The nature, history and current characteristics of Language Engineering are discussed, which is contrasted with Natural Language Processing and Computational Linguistics, and which is shown to have attained its own distinct identity in recent years.
Open Mind Word Expert: Creating Large Annotated Data Collections with Web Users’ Help
43 Citations2003Rada Mihalcea, Timothy Chklovski
Open Mind Word Expert is an implemented active learning system that aims to create large annotated corpora by tapping into the world's vast pool of knowledge by relying on the vast number of Web users who contribute their knowledge to data annotation.
Callisto: A Configurable Annotation Workbench
41 Citations2004David Day, Chad McHenry +2 more
Callisto provides a facility for the independent development, compilation and installation of task module plug-ins (in the form of Java Archive jar files) and provides a set of annotation services to which all separate GUI components can subscribe, enabling a common framework through which annotation updates are propagated to all components.
The Kluwer international series on information retrievalUsing Hedges to Classify Citations in Scientific Articles
36 Citations2006Chrysanne Di Marco, Frederick Kroon +1 more
It is conjecture that hedging is a significant aspect of the rhetorical structure of citation contexts and that the pragmatics of hedges may help in determining the rhetorical purpose of citations.
Recognising affect in text using pointwise-mutual information
35 Citations2010Jonathon Read
Experiments conducted to evaluate an algorithm that attempts to automatically recognise emotions (affect) in written language indicate that the method is making slightly informed choices, and could be useful as part of a holistic approach to recognising the affect represented in text.