Assessing the Accuracy of Automated Twitter Sentiment Coding

Overall, automated sentiment coding appears to have limited reliability and appears to be accurately accomplished only for very simple statements in which a keyword is used to convey its typical meaning, while Neutral statements appear to be problematic for some, but not all, companies.

Abstract

ABSTRACT Social media have provided consumers with numerous outlets for disseminating their brand-related comments. Given impact of these comments on brand image and brand success, marketers have begun to rely on third-party companies to automatically track, collect and analyze these comments with regard to their content and sentiment. There is, however, an absence of controlled research which objectively and systematically assesses accuracy of these companies' automated sentiment coding. This research evaluated automated sentiment coding accuracy and misclassification errors of six leading third-party companies for a broad range of comment types and forms. Overall, automated sentiment coding appears to have limited reliability and appears to be accurately accomplished only for very simple statements in which a keyword used to convey its typical meaning. Statements without keywords or statements in which keyword meaning reversed through negation or context are accurately coded at very low levels. Neutral statements appear to be problematic for some, but not all, companies. Implications of research for use of automated sentiment analysis for brand decisionmaking are presented. INTRODUCTION Marketers' attempt to manage brand image in pre-social media era was straightforward. Brand managers would decide what they wanted consumers to think about their brand and then, through advertising and other consumer-directed communications, would tightly control messages to which consumers were exposed. Ideally, these communications would create or reinforce desired brand image. Communications planning took perspective of ?? tell you things that we think are really important [and what you should think] about our brand and (Bostic, 2010) as consumers had little opportunity to disseminate their own perspective. Social media have fundamentally altered how brand image created, maintained and changed. Today's successful marketers understand that social media have reduced their control and they realize that they are but one (albeit important) contributor to a dialogue about brand. These marketers realize that their brand's image will not only be result of what they say on their brand's own behalf, but additionally, what consumers say and how they interact with and respond to these consumers' comments (Davenport & Beck, 2002; Esch et al., 2006). Consumers are now in at least a shared leadership position with regard to brand image creation, in essence telling marketers your brand whatever [we] say it is (Li & Bernoff, 2008). Li and Bernoff (2008) refer to transfer of power from businesses to individuals as the groundswell. Examples of groundswell are abundance of online brand and product reviews as well as online communities such as Get Satisfaction (http://www. getsatisfaction.com), which provides a forum for customers to raise questions or complain about a wide range of companies, and for resulting discussions to be displayed for other consumers to search and view. Online product reviews and sites such as Get Satisfaction can have a profound effect on brand image and brand success given that consumer decision-making process greatly influenced by other consumers' comments (Goldberg et al., 2001; Kelsey Group, 2007; Jansen et al., 2009), especially at time of product evaluation and purchase (Zabin and Jefferies, 2008). Clearly, it important for marketers to understand what consumers are saying about their brands and products. Specifically, marketers need to be aware of what consumers are saying (i.e., specific comments about brand) and they need to understand what consumers are feeling (i.e., sentiment associated with a brand comment, that is, whether a particular comment positive, negative or neutral). Cai et al. (2010) provide a rationale for specific focus on sentiment monitoring: The voice of web to gain consumer, brand and market insights can be truly differentiating and valuable to today's corporations [and] one important form of insights can be derived from sentiment analysis. …