Yahoo! for Amazon: Sentiment Parsing from Small Talk on the Web

Abstract

The internet has made it feasible to tap a continuous stream of public from the world wide web, quite literally permitting one to feel the pulse of any issue under consideration. We present a methodology for real time extraction in the domain of finance. With the advent of the web, there has been a sharp increase in the influence of individuals on the stock market via web-based trading and the posting of to stock message boards. While it is importantto capture this sentiment of small investors, as yet, no index of has been compiled. This paper comprises (a) a technology for extracting small investor from web sources to create an index, and (b) illustrative applications of the methodology. We make use of computerized natural language and statistical algorithms for the automated classification of messages posted on the web. We design a suite of classification algorithms, each of different theoretical content, with a view to characterizing the of any single posting to a message board. The use of multiple methods allows imposition of voting rules in the classification process. It also enables elimination of fuzzy messages which are better off uninterpreted. A majority rule across algorithms vastly improves classification accuracy, but also leads to a natural increase in the number of messages classified as fuzzy. The classifier achieves an accuracy of 62% (versus a random classification accuracy of 33%), and compares favorably against human agreement on message classification, which was 72%. The technology is computationally efficient, allowing the access and interpretations of thousands of messages within minutes. Our illustrative applications show evidence of a strong link between market movements and sentiment. Based on approximately 25,000 messages for the last quarter of 2000, we found evidence that is based on stock movements.