SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods
EPJ Data SciencePublished 7 July 2016Open access
Filipe Nunes Ribeiro, Matheus Araújo, Pollyanna Gonçalves, Marcos André Gonçalves, Fabrício Benevenuto
Citations474
SJR quartileQ1
SJR score0.74
SNIP1.29
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A benchmark comparison of twenty-four popular sentiment analysis methods, covering messages posted on social networks, movie and product reviews, as well as opinions and comments in news articles is presented, highlighting the extent to which the prediction performance of these methods varies considerably across datasets.
Abstract
Оскільки в наш час від імені науки відбувається легітимація і канонізація \nнайбільш респектабельного знання, то цілком закономірно, що мова науки \nзокрема і знання загалом на рівні стереотипів має ледве не сакральний статус. \nНаскільки такий стан речей є коректним? Сформулювати переконливу \nвідповідь дозволяє дослідження аспекту соціокультурної зумовленості \nнаукового знання.
Keywords
Computer Science
BiometricsThe Measurement of Observer Agreement for Categorical Data
78,934 Citations1977J. Richard Landis, Gary G. Koch
A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
Journal of Personality and Social PsychologyDevelopment and validation of brief measures of positive and negative affect: The PANAS scales.
31,665 Citations1988David Watson, Lee A. Clark +1 more
IEEE Transactions on Evolutionary ComputationNo free lunch theorems for optimization
13,936 Citations1997David H. Wolpert, William G. Macready
A framework is developed to explore the connection between effective optimization algorithms and the problems they are solving and a number of "no free lunch" (NFL) theorems are presented which establish that for any algorithm, any elevated performance over one class of problems is offset by performance over another class.
Mining and summarizing customer reviews
7,714 Citations2004Minqing Hu, Bing Liu
This research aims to mine and to summarize all the customer reviews of a product, and proposes several novel techniques to perform these tasks.
Thumbs up?
6,987 Citations2002Bo Pang, Lillian Lee +1 more
This work considers the problem of classifying documents not by topic, but by overall sentiment, e.g., determining whether a review is positive or negative, and concludes by examining factors that make the sentiment classification problem more challenging.
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
6,774 Citations2013Richard Socher, Alex Perelygin +5 more
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
Journal of Language and Social PsychologyThe Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
5,776 Citations2009Yla Tausczik, James W. Pennebaker
Empirical results using LIWC demonstrate its ability to detect meaning in a wide variety of experimental settings, including to show attentional focus, emotionality, social relationships, thinking styles, and individual differences.
Proceedings of the International AAAI Conference on Web and Social MediaVADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text
5,749 Citations2014Cecelia Hutto, Éric Gilbert
Interestingly, using the authors' parsimonious rule-based model to assess the sentiment of tweets, it is found that VADER outperforms individual human raters, and generalizes more favorably across contexts than any of their benchmarks.
Journal of Computational ScienceTwitter mood predicts the stock market
4,995 Citations2011Johan Bollen, Huina Mao +1 more
This work investigates whether measurements of collective mood states derived from large-scale Twitter feeds are correlated to the value of the Dow Jones Industrial Average (DJIA) over time and indicates that the accuracy of DJIA predictions can be significantly improved by the inclusion of specific public mood dimensions but not others.
Proceedings of the National Academy of SciencesExperimental evidence of massive-scale emotional contagion through social networks
3,658 Citations2014Adam Kramer, Jamie Guillory +1 more
Emotions expressed by others on Facebook influence the authors' own emotions, constituting experimental evidence for massive-scale contagion via social networks and suggesting that the observation of others’ positive experiences constitutes a positive experience for people.
A Convolutional Neural Network for Modelling Sentences
3,561 Citations2014Nal Kalchbrenner, Edward Grefenstette +1 more
A convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) is described that is adopted for the semantic modelling of sentences and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations.
Recognizing contextual polarity in phrase-level sentiment analysis
3,379 Citations2005Theresa Wilson, Janyce Wiebe +1 more
A new approach to phrase-level sentiment analysis is presented that first determines whether an expression is neutral or polar and then disambiguates the polarity of the polar expressions.
A sentimental education
3,343 Citations2004Bo Pang, Lillian Lee
A novel machine-learning method is proposed that applies text-categorization techniques to just the subjective portions of the document, which greatly facilitates incorporation of cross-sentence contextual constraints.
Encyclopedia of Machine Learning and Data MiningSentiment Analysis and Opinion Mining
3,319 Citations2017Lei Zhang, Bing Liu
Computational LinguisticsLexicon-Based Methods for Sentiment Analysis
3,255 Citations2011Maite Taboada, Julian Brooke +3 more
The Semantic Orientation CALculator (SO-CAL) uses dictionaries of words annotated with their semantic orientation (polarity and strength), and incorporates intensification and negation, and is applied to the polarity classification task.
Synthesis lectures on human language technologiesSentiment Analysis and Opinion Mining
3,241 Citations2012Bing Liu
This book is a comprehensive introductory and survey text that covers all important topics and the latest developments in the field with over 400 references and is suitable for students, researchers and practitioners who are interested in social media analysis in general and sentiment analysis in particular.
Proceedings of the International AAAI Conference on Web and Social MediaMeasuring User Influence in Twitter: The Million Follower Fallacy
3,009 Citations2010Meeyoung Cha, Hamed Haddadi +2 more
An in-depth comparison of three measures of influence, using a large amount of data collected from Twitter, is presented, suggesting that topological measures such as indegree alone reveals very little about the influence of a user.
Behavior Research MethodsMoving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English
2,833 Citations2009Marc Brysbaert, Boris New
The size of the corpus, the language register on which the corpus is based, and the definition of the frequency measure were investigated, finding that lemma frequencies are not superior to word form frequencies in English and that a measure of contextual diversity is better than a measure based on raw frequency of occurrence.
Proceedings of the International AAAI Conference on Web and Social MediaPredicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
2,681 Citations2010Andranik Tumasjan, Timm O. Sprenger +2 more
It is found that the mere number of messages mentioning a party reflects the election result, and joint mentions of two parties are in line with real world political ties and coalitions.
SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining.
2,652 Citations2010Stefano Baccianella, Andrea Esuli +1 more
This work discusses SENTIWORDNET 3.0, a lexical resource explicitly devised for supporting sentiment classification and opinion mining applications, and reports on the improvements concerning aspect (b) that it embodies with respect to version 1.0.
Computational IntelligenceCROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON
2,584 Citations2012Saif M. Mohammad, Peter D. Turney
It is shown how the combined strength and wisdom of the crowds can be used to generate a large, high‐quality, word–emotion and word–polarity association lexicon quickly and inexpensively.
SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining
2,489 Citations2006Andrea Esuli, Fabrizio Sebastiani
SENTIWORDNET is a lexical resource in which each WORDNET synset is associated to three numerical scores Obj, Pos and Neg, describing how objective, positive, and negative the terms contained in the synset are.
Choice Reviews OnlineThe art of computer systems performance analysis: techniques for experimental design, measurement, simulation, and modeling
2,109 Citations1992Robert Y. Al-Jaar
Behavior Research MethodsNorms of valence, arousal, and dominance for 13,915 English lemmas
1,978 Citations2013Amy Beth Warriner, Victor Kuperman +1 more
This work extended the ANEW database to nearly 14,000 English lemmas, providing researchers with a much richer source of information, including gender, age, and educational differences in emotion norms.
Cheap and fast---but is it good?
1,906 Citations2008Rion Snow, Brendan O’Connor +2 more
This work explores the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web, and proposes a technique for bias correction that significantly improves annotation quality on two tasks.
Computers and the HumanitiesAnnotating Expressions of Opinions and Emotions in Language
1,770 Citations2005Janyce Wiebe, Theresa Wilson +1 more
The manual annotation process and the results of an inter-annotator agreement study on a 10,000-sentence corpus of articles drawn from the world press are presented.
Communications of the ACMTechniques and applications for sentiment analysis
1,494 Citations2013Ronen Feldman
The main applications and challenges of one of the hottest research areas in computer science are revealed.
Proceedings of the International AAAI Conference on Web and Social MediaTwitter Sentiment Analysis: The Good the Bad and the OMG!
1,243 Citations2021Efthymios Kouloumpis, Theresa Wilson +1 more
Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification
1,167 Citations2014Duyu Tang, Furu Wei +4 more
Three neural networks are developed to effectively incorporate the supervision from sentiment polarity of text (e.g. sentences or tweets) in their loss functions and the performance of SSWE is improved by concatenating SSWE with existing feature set.
Proceedings of the International AAAI Conference on Web and Social MediaModeling Public Mood and Emotion: Twitter Sentiment and Socio-Economic Phenomena
950 Citations2021Johan Bollen, Huina Mao +1 more
It is speculated that large scale analyses of mood can provide a solid platform to model collective emotive trends in terms of their predictive value with regards to existing social as well as economic indicators.
Effective Use of Word Order for Text Categorization with Convolutional Neural Networks
776 Citations2015Rie Johnson, Tong Zhang
A straightforward adaptation of CNN from image to text, a simple but new variation which employs bag-of-word conversion in the convolution layer is proposed and an extension to combine multiple convolution layers is explored for higher accuracy.
SemEval-2007 task 14
599 Citations2007Carlo Strapparava, Rada Mihalcea
The data set used in the evaluation and the results obtained by the participating systems are described, meant as an exploration of the connection between emotions and lexical semantics.
International Conference on Weblogs and Social MediaLarge-Scale Sentiment Analysis for News and Blogs
558 Citations2007Namrata Godbole, Manjunath Srinivasaiah +1 more
A System for Real-time Twitter Sentiment Analysis of 2012 U.S. Presidential Election Cycle
551 Citations2012Hao Wang, Doğan Can +3 more
A system for real-time analysis of public sentiment toward presidential candidates in the 2012 U.S. election as expressed on Twitter, a micro-blogging service, offers a new and timely perspective on the dynamics of the electoral process and public opinion.
arXiv (Cornell University)A new ANEW: Evaluation of a word list for sentiment analysis in\n microblogs
537 Citations2011Finn Årup Nielsen
Characterizing debate performance via aggregated twitter sentiment
473 Citations2010Nicholas Diakopoulos, David A. Shamma
An analytical methodology and visual representations are developed that could help a journalist or public affairs person better understand the temporal dynamics of sentiment in reaction to the debate video.
OpinionFinder
471 Citations2005Theresa Wilson, Paul Hoffmann +7 more
OpinionFinder is a system that performs subjectivity analysis, automatically identifying when opinions, sentiments, speculations, and other private states are present in text.
Proceedings of the 32nd annual meeting on Association for Computational Linguistics
470 Citations1994James Pustejovsky
arXiv (Cornell University)NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets
460 Citations2013Saif M. Mohammad, Svetlana Kiritchenko +1 more
University of Maribor digital library (University of Maribor)Basic Business Statistics: Concepts and Applications
449 Citations2011Mark L. Berenson, Левин
This comprehensive and innovative study on business statistics uses a practical, data-analytic approach and focuses on data analysis and interpretation of computer output with a reduced focus on hand calculations.
Data Mining and Knowledge DiscoverySurvey on mining subjective data on the web
441 Citations2011Mikalai Tsytsarau, Themis Palpanas
The development of Sentiment Analysis and Opinion Mining during the last years are reviewed, and the evolution of a relatively new research direction is discussed, namely, Contradiction Analysis.
Proceedings of the AAAI Conference on Artificial IntelligenceSenticNet 3: A Common and Common-Sense Knowledge Base for Cognition-Driven Sentiment Analysis
439 Citations2014Erik Cambria, Daniel Olsher +1 more
SenticNet 3 models nuanced semantics and sentics (that is, the conceptual and affective information associated with multi-word natural language expressions), representing information with a symbolic opacity of an intermediate nature between that of neural networks and typical symbolic systems.
SemEval-2016 Task 4: Sentiment Analysis in Twitter
433 Citations2016Preslav Nakov, Sara Rosenthal +3 more
The fourth year of the SemEval-2016 Task 4 comprises five subtasks, three of which represent a significant departure from previous editions, and the task continues to be very popular, attracting a total of 43 teams.
Proceedings of the National Academy of SciencesHuman language reveals a universal positivity bias
403 Citations2015Peter Sheridan Dodds, Eric M. Clark +12 more
Using human evaluation of 100,000 words spread across 24 corpora in 10 languages diverse in origin and culture, evidence of a deep imprint of human sociality in language is presented, observing that the words of natural human language possess a universal positivity bias.
arXiv (Cornell University)Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena
318 Citations2009Johan Bollen, Alberto Pepe +1 more
Joint Conference on Lexical and Computational SemanticsEmotional Tweets
299 Citations2012Saif M. Mohammad
This paper describes how a Twitter emotion corpus is created from Twitter posts using emotion-word hashtags, and extracts a word-emotion association lexicon that leads to significantly better results than the manually crafted WordNet Affect lexicon in an emotion classification task.
Measuring the Happiness of Large-Scale Written Expression: Songs, Blogs, and Presidents
282 Citations2015Peter Sheridan Dodds, Christopher M. Danforth +1 more
Journal of Happiness StudiesMeasuring the Happiness of Large-Scale Written Expression: Songs, Blogs, and Presidents
261 Citations2009Peter Sheridan Dodds, Christopher M. Danforth
By incorporating direct human assessment of words, this work quantifies happiness levels on a continuous scale for a diverse set of large-scale texts: song titles and lyrics, weblogs, and State of the Union addresses.
Stirling Online Research Repository (University of Stirling)SenticNet: A Publicly Available Semantic Resource for Opinion Mining
254 Citations2010Erik Cambria, Robert E. Speer +2 more
SenticNet is a publicly available resource for opinion mining built exploiting AI and Semantic Web techniques and uses dimensionality reduction to infer the polarity of common sense concepts and hence provide a public resource for mining opinions from natural language text at a semantic, rather than just syntactic, level.
Journal of Machine Learning ResearchPattern for python
240 Citations2012Tom De Smedt, Walter Daelemans
Pattern is a package for Python 2.4+ with functionality for web mining, natural language processing, naturallanguage processing, machine learning, and network analysis for graph centrality and visualization.
Generating high-coverage semantic orientation lexicons from overtly marked words and a thesaurus
237 Citations2009Saif M. Mohammad, Cody Dunne +1 more
This work proposes a simple approach to generate a high-coverage semantic orientation lexicon, which includes both individual words and multi-word expressions, using only a Roget-like thesaurus and a handful of affixes and has properties that support the Polyanna Hypothesis.
Understanding complex systemsThe Heart and Soul of the Web? Sentiment Strength Detection in the Social Web with SentiStrength
193 Citations2016Mike Thelwall
This chapter describes the sentiment strength detection program SentiStrength that was developed during the CyberEmotions project to detect the strength of sentiments expressed in social web texts.
Language Resources and EvaluationMethods for Creating Semantic Orientation Dictionaries.
152 Citations2006Maite Taboada, Caroline Anthony +1 more
Different methods for creating a dictionary of words with their corresponding semantic orientation (SO) are described and compared, and how well different dictionaries helped determine the SO of entire texts is tested.
EPJ Data SciencePositive words carry less information than negative words
121 Citations2012David García, Antonios Garas +1 more
Taking into account the frequency of word usage, it is found that words with a positive emotional content are more frequently used, which lends support to Pollyanna hypothesis that there should be a positive bias in human expression.
Lecture notes in computer scienceOn the Predictability of Stock Market Behavior Using StockTwits Sentiment and Posting Volume
117 Citations2013Nuno Oliveira, Paulo Cortez +1 more
There is evidence that posting volume can improve the forecasts of trading volume, which is useful for measuring stock liquidity (e.g. assets easily sold), and no evidence of return predictability using sentiment indicators, and of information content of posting volume for forecasting volatility are found.
arXiv (Cornell University)Breaking the News: First Impressions Matter on Online News
110 Citations2015Júlio C. S. Reis, Fabrício Benevenuto +4 more
Characterizing Smoking and Drinking Abstinence from Social Media
98 Citations2015Acar Tamersoy, Munmun De Choudhury +1 more
This paper uses the self-reported "badge" information of nearly a thousand users as gold standard information on their abstinence status to characterize long-term abstinence and builds supervised learning based statistical models that use the linguistic features of the content shared by users as well as the network structure of their social interactions.
Proceedings of the International AAAI Conference on Web and Social MediaTweetin' in the Rain: Exploring Societal-Scale Effects of Weather on Mood
89 Citations2021Anikó Hannák, Eric Anderson +4 more
Using machine learning techniques on the Twitter corpus correlated with the weather at the time and location of the tweets, it is found that aggregate sentiment follows distinct climate, temporal, and seasonal patterns.
Sentiment analysis of user comments for one-class collaborative filtering over ted talks
78 Citations2013Nikolaos Pappas, Andréi Popescu-Belis
A sentiment-aware nearest neighbor model (SANN) for multimedia recommendations over TED talks, which makes use of user comments is proposed, which outperforms significantly, by more than 25% on unseen data, several competitive baselines.
Proceedings of the International AAAI Conference on Web and Social MediaBreaking the News: First Impressions Matter on Online News
72 Citations2021Julio Rieis, Fabrício De Souza +4 more
It is discovered that the sentiment of the headline is strongly related to the popularity of the news and also with the dynamics of the posted comments on that particular news.
arXiv (Cornell University)PANAS-t: A Pychometric Scale for Measuring Sentiments on Twitter
39 Citations2013Pollyanna Gonçalves, Fabrício Benevenuto +1 more
This paper proposes PANAS-t, which measures sentiments from short text updates in Twitter based on a well-established psychometric scale, PANAS (Positive and Negative Affect Schedule), and demonstrates that it can efficiently capture the expected sentiments of a wide variety of issues spanning tragedies, technology releases, political debates, and healthcare.
Umigon: sentiment analysis for tweets based on terms lists and heuristics
29 Citations2013Clement Levallois
The “sentiment detection engine” of Umigon consists of 4 main parts, which are detailed below: detection of semantic features in the entire tweet, evaluation of hashtags, global heuristics, and evaluation of hashtags.
Proceedings of the International AAAI Conference on Web and Social MediaiFeel 2.0: A Multilingual Benchmarking System for Sentence-Level Sentiment Analysis
21 Citations2021Matheus da Silva Araújo, João P. Diniz +6 more
iFeel 2.0 is proposed, an online web system that implements 19 sentence-level sentiment analysis methods and allows users to easily label a dataset with all of them and incorporates a multiple language feature to allow methods designed for specific languages to be easily compared with a baseline approach that simply translates the input data to English.
Lecture notes in computer scienceDistinguishing the Popularity between Topics: A System for Up-to-Date Opinion Retrieval and Mining in the Web
18 Citations2013Nikolaos Pappas, Georgios Katsimpras +1 more
An online system for opinion mining and retrieval that is able to discover up-to-date web pages on given topics using focused crawling agents, extract opinionated textual parts from web pages, and estimate their polarity using opinion mining agents is presented.
Proceedings of the International AAAI Conference on Web and Social MediaMagnet News: You Choose the Polarity of What You Read
10 Citations2014Júlio C. S. Reis, Pollyanna Gonçalves +3 more
The Magnet News is presented, a Web tool in which users can choose if they want to see positive or negative news, and monitors the sentiment of news from important newspapers using the SentiStrength, a sentiment method proposed by literature that has been proved to be efficient in previous analysis.