Method for evaluating sentiment analysis tools
Published 1 January 2013
V. Poot
Citations1
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is concluded that the method provided is useable for laymen in order to critically review a sentiment analysis tool and that developers of sentiment analysis tools need to take biases and their transparency into account when developing a sentimentAnalysis tools.
Abstract
Sentiment analysis tools are tools that extract subjective information from texts and process it into valuable information. A lot of these tools are available for individual users or small organizations (the laymen), who might not be familiar with these tools. This paper will start with a literature study on epistemological traditions and sentiment analysis for establishing criteria to evaluate the value of knowledge generated by sentiment analysis tools. Next a method for laymen to use these criteria for evaluating sentiment analysis tools is provided together with an evaluation of several available tools. This paper concludes that the method provided is useable for laymen in order to critically review a sentiment analysis tool and that developers of sentiment analysis tools need to take biases and their transparency into account when developing a sentiment analysis tools. Ultimately, recommendations for further research are given.
Keywords
Computer Science
BiometricsThe Measurement of Observer Agreement for Categorical Data
78,934 Citations1977J. Richard Landis, Gary G. Koch
A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
Teaching SociologyThe Practice of Social Research
14,440 Citations1989Alan Neustadtl, Earl R. Babbie
This chapter discusses the construction of Inquiry, the science of inquiry, and the role of data in the design of research.
now publishers, Inc. eBooksOpinion Mining and Sentiment Analysis
6,745 Citations2008Bo Pang, Lillian Lee
Journal of Computational ScienceTwitter mood predicts the stock market
4,995 Citations2011Johan Bollen, Huina Mao +1 more
This work investigates whether measurements of collective mood states derived from large-scale Twitter feeds are correlated to the value of the Dow Jones Industrial Average (DJIA) over time and indicates that the accuracy of DJIA predictions can be significantly improved by the inclusion of specific public mood dimensions but not others.
IEEE ExpertData mining and knowledge discovery: making sense out of data
4,643 Citations1996U.M. Feyyad
Find loads of the data mining and knowledge discovery making sense out of data book catalogues in this site as the choice of you visiting this page.
A holistic lexicon-based approach to opinion mining
1,388 Citations2008Xiaowen Ding, Bing Liu +1 more
This paper proposes a holistic lexicon-based approach to solving the problem of determining the semantic orientations (positive, negative or neutral) of opinions expressed on product features in reviews by exploiting external evidences and linguistic conventions of natural language expressions.
Journal of the American Society for Information Science and TechnologySentiment strength detection for the social web
1,063 Citations2011Mike Thelwall, Kevan Buckley +1 more
An improved version of the algorithm SentiStrength for sentiment strength detection across the social web that primarily uses direct indications of sentiment is assessed, suggesting that, even unsupervised, Senti strength is robust enough to be applied to a wide variety of different social web contexts.
Joint sentiment/topic model for sentiment analysis
972 Citations2009Chenghua Lin, Yulan He
A novel probabilistic modeling framework based on Latent Dirichlet Allocation (LDA), called joint sentiment/topic model (JST), which detects sentiment and topic simultaneously from text is proposed, which is fully unsupervised.
ACM Transactions on Information SystemsSentiment analysis in multiple languages
894 Citations2008Ahmed Abbasi, Hsinchun Chen +1 more
Stylistic features significantly enhanced performance across all testbeds while EWGA also outperformed other feature selection methods, indicating the utility of these features and techniques for document-level classification of sentiments.
Journal of InformetricsSentiment analysis: A combined approach
791 Citations2009Rudy Prabowo, Mike Thelwall
This paper combines rule-based classification, supervised learning and machine learning into a new combined method, and proposes a semi-automatic, complementary approach in which each classifier can contribute to other classifiers to achieve a good level of effectiveness.
Sentiment analyzer: extracting sentiments about a given topic using natural language processing techniques
720 Citations2004Junbo Yi, Tetsuya Nasukawa +2 more
This work presents sentiment analyzer (SA) that extracts sentiment (or opinion) about a subject from online text documents using natural language processing (NLP) techniques.
Using appraisal groups for sentiment analysis
562 Citations2005Casey Whitelaw, Navendu Garg +1 more
A new method for sentiment classification based on extracting and analyzing appraisal groups such as ``very good'' or ``not terribly funny'' is presented, based on several task-independent semantic taxonomies based on Appraisal Theory.
Decision Support SystemsUsing text mining and sentiment analysis for online forums hotspot detection and forecast
478 Citations2009Nan Li, Desheng Wu
This paper creates an algorithm to automatically analyze the emotional polarity of a text and to obtain a value for each piece of text, which is combined with K-means clustering and support vector machine (SVM) to develop unsupervised text mining approach.
Data Mining and Knowledge DiscoverySurvey on mining subjective data on the web
441 Citations2011Mikalai Tsytsarau, Themis Palpanas
The development of Sentiment Analysis and Opinion Mining during the last years are reviewed, and the evolution of a relatively new research direction is discussed, namely, Contradiction Analysis.
Service BusinessUser generated content: the use of blogs for tourism organisations and tourism consumers
407 Citations2008Gary Akehurst
The development of user generated content and specifically the use of web logs or blogs is explored; there is a need to locate, extract and interpret blog content and this has proven so far to be time consuming, exhausting and costly, thus negating the relative value of the information obtained.
Decision Support SystemsMining comparative opinions from customer reviews for Competitive Intelligence
302 Citations2010Kaiquan Xu, Stephen Shaoyi Liao +2 more
A novel graphical model is proposed to extract and visualize comparative relations between products from customer reviews, with the interdependencies among relations taken into consideration, to help enterprises discover potential risks and further design new products and marketing strategies.
How (Not) to Predict Elections
284 Citations2011Panagiotis Metaxas, Eni Mustafaraj +1 more
It is found that electoral predictions using the published research methods on Twitter data are not better than chance and a set of standards that any theory aiming to predict elections (or other social events) using social media should follow is proposed.
Decision Support SystemsQuality evaluation of product reviews using an information quality framework
274 Citations2010Chien Chin Chen, You-De Tseng
This paper treats the evaluation of review quality as a classification problem and employs an effective information quality framework to extract representative review features and demonstrates that the proposed method outperforms state-of-the-art approaches significantly.
OpinionMiner
267 Citations2009Wei Jin, Hung Hay Ho +1 more
The OpinionMiner system designed in this work aims to mine customer reviews of a product and extract high detailed product entities on which reviewers express their opinions.
Stirling Online Research Repository (University of Stirling)SenticNet: A Publicly Available Semantic Resource for Opinion Mining
254 Citations2010Erik Cambria, Robert E. Speer +2 more
SenticNet is a publicly available resource for opinion mining built exploiting AI and Semantic Web techniques and uses dimensionality reduction to infer the polarity of common sense concepts and hence provide a public resource for mining opinions from natural language text at a semantic, rather than just syntactic, level.
Decision Support SystemsPredicting consumer sentiments from online text
236 Citations2010Xue Bai
A heuristic search-enhanced Markov blanket model is proposed that is able to capture the dependencies among words and provide a vocabulary that is adequate for the purpose of extracting sentiments.
How opinions are received by online communities
236 Citations2009Cristian Danescu-Niculescu-Mizil, Gueorgi Kossinets +2 more
It is found that the perceived helpfulness of a review depends not just on its content but also but also in subtle ways on how the expressed evaluation relates to other evaluations of the same product.
Show me the money!
226 Citations2007Nikolay Archak, Anindya Ghose +1 more
A novel hybrid technique combining text mining and econometrics that models consumer product reviews as elements in a Tensor product of feature and evaluation spaces is developed and impute the quantitative impact of consumer reviews on product demand as a linear functional from this tensor product space.
Expert Systems with ApplicationsGather customer concerns from online product reviews – A text summarization approach
193 Citations2008Jiaming Zhan, Han Tong Loh +1 more
This work proposes an automatic summarization approach based on the analysis of review articles' internal topic structure to assemble customer concerns and shows that the proposed approach outperforms the peer approaches, i.e. opinion mining and clustering-summarization, in terms of users' responsiveness and its ability to discover the most important topics.
IEEE Transactions on Visualization and Computer GraphicsOpinionSeer: Interactive Visualization of Hotel Customer Feedback
176 Citations2010Yingcai Wu, Furu Wei +5 more
OpinionSeer is an interactive visualization system that could visually analyze a large collection of online hotel customer reviews and is built on a new visualization-centric opinion mining technique that considers uncertainty for faithfully modeling and analyzing customer opinions.
A generation model to unify topic relevance and lexicon-based sentiment for opinion retrieval
129 Citations2008Min Zhang, Xingyao Ye
A novel generation model that unifies topic-relevance and opinion generation by a quadratic combination is proposed and demonstrates that in the opinion retrieval task, a Bayesian approach to combining multiple ranking functions is superior to using a linear combination.
Emerging topic detection using dictionary learning
127 Citations2011Shiva Prasad Kasiviswanathan, Prem Melville +2 more
This work addresses the problem of identifying emerging topics through the use of dictionary learning by proposing a two stage approach respectively based on detection and clustering of novel user-generated content and derives a scalable approach by using the alternating directions method to solve the resulting optimization problems.
Sentiment Mining in WebFountain
97 Citations2005Junbo Yi, W. Niblack
This paper describes the fully functional system environment and the algorithms, and reports the performance of the sentiment miner, a sentiment miner that determines sentiment of each subject reference using natural language processing techniques.
Opinion mining in legal blogs
97 Citations2007Jack G. Conrad, Frank Schilder
A survey into the scope and utility of opinion mining in legal Weblogs (a.k.a. blawgs) is performed and a language modeling approach deployed for both subjectivity analysis and polarity analysis is examined.
Lecture notes in computer scienceSenticSpace: Visualizing Opinions and Sentiments in a Multi-dimensional Vector Space
67 Citations2010Erik Cambria, Amir Hussain +2 more
This work builds a knowledge base which merges common sense and affective knowledge and visualize it in a multi-dimensional vector space, which it is called SenticSpace.
Transactions of the Japanese Society for Artificial IntelligenceOpinion Mining from Web Documents: Extraction and Structurization
46 Citations2007Nozomi Kobayashi, Kentaro Inui +1 more
How customer reviews in web documents can be best structured is discussed and a method for extracting opinions that represent evaluation of concumer products in a structured form is developed.
Preface
9 Citations2007