New Word Detection for Sentiment Analysis

This paper proposes a fully unsupervised, purely data-driven framework for extracting new sentiment words from large-scale user-generated content and designs statistical measures respectively to quantify the utility of a lexical pattern and to measure the possibility of a word being a new word.

Abstract

Automatic extraction of new words is an indispensable precursor to many NLP tasks such as Chinese word segmentation, named entity extraction, and sentiment analysis. This paper aims at extracting new sentiment words from large-scale user-generated content. We propose a fully unsupervised, purely data-driven framework for this purpose. We design statistical measures respectively to quantify the utility of a lexical pattern and to measure the possibility of a word being a new word. The method is almost free of linguistic resources (except POS tags), and requires no elaborated linguistic rules. We also demonstrate how new sentiment word will benefit sentiment analysis. Experiment results demonstrate the effectiveness of the proposed method.