Combination of Convolutional and Recurrent Neural Network for Sentiment Analysis of Short Texts
International Conference on Computational LinguisticsPublished 1 December 2016
Xingyou Wang, Weijie Jiang, Zhiyong Luo
Citations273
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A jointed CNN and RNN architecture is described, taking advantage of the coarse-grained local features generated by CNN and long-distance dependencies learned via RNN for sentiment analysis of short texts.
Abstract
Sentiment analysis of short texts is challenging because of the limited contextual information they usually contain. In recent years, deep learning models such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been applied to text sentiment analysis with comparatively remarkable results. In this paper, we describe a jointed CNN and RNN architecture, taking advantage of the coarse-grained local features generated by CNN and long-distance dependencies learned via RNN for sentiment analysis of short texts. Experimental results show an obvious improvement upon the state-of-the-art on three benchmark corpora, MR, SST1 and SST2, with 82.28%, 51.50% and 89.95% accuracy, respectively.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Convolutional Neural Networks for Sentence Classification
13,798 Citations2014Yoon Kim
The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification, and are proposed to allow for the use of both task-specific and static vectors.
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
arXiv (Cornell University)Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
10,728 Citations2014Jun‐Young Chung, Çaǧlar Gülçehre +2 more
These advanced recurrent units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU), are found to be comparable to LSTM.
IEEE Transactions on Neural NetworksLearning long-term dependencies with gradient descent is difficult
8,461 Citations1994Yoshua Bengio, P. Simard +1 more
This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods.
Mining and summarizing customer reviews
7,714 Citations2004Minqing Hu, Bing Liu
This research aims to mine and to summarize all the customer reviews of a product, and proposes several novel techniques to perform these tasks.
Thumbs up?
6,987 Citations2002Bo Pang, Lillian Lee +1 more
This work considers the problem of classifying documents not by topic, but by overall sentiment, e.g., determining whether a review is positive or negative, and concludes by examining factors that make the sentiment classification problem more challenging.
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
6,774 Citations2013Richard Socher, Alex Perelygin +5 more
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
6,616 Citations2014Kyunghyun Cho, Bart van Merriënboer +2 more
It is shown that the neural machine translation performs relatively well on short sentences without unknown words, but its performance degrades rapidly as the length of the sentence and the number of unknown words increase.
Meeting of the Association for Computational LinguisticsThumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews
3,654 Citations2002Peter Peter, Turney
A Convolutional Neural Network for Modelling Sentences
3,561 Citations2014Nal Kalchbrenner, Edward Grefenstette +1 more
A convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) is described that is adopted for the semantic modelling of sentences and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations.
Seeing stars
2,127 Citations2005Bo Pang, Lillian Lee
A meta-algorithm is applied, based on a metric labeling formulation of the rating-inference problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels.
Semantic Compositionality through Recursive Matrix-Vector Spaces
1,297 Citations2012Richard Socher, Brody Huval +2 more
A recursive neural network model that learns compositional vector representations for phrases and sentences of arbitrary syntactic type and length and can learn the meaning of operators in propositional logic and natural language is introduced.
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
1,198 Citations2011Richard Socher, Jeffrey Pennington +3 more
A novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions that outperform other state-of-the-art approaches on commonly used datasets, without using any pre-defined sentiment lexica or polarity shifting rules.
International Conference on Computational LinguisticsDeep Convolutional Neural Networks for Sentiment Analysis of Short Texts
1,191 Citations2014Cícero dos Santos, Maíra Gatti de Bayser
A new deep convolutional neural network is proposed that exploits from characterto sentence-level information to perform sentiment analysis of short texts and achieves state-of-the-art results for single sentence sentiment prediction.
arXiv (Cornell University)Character-Aware Neural Language Models
1,022 Citations2015Yoon Kim, Yacine Jernite +2 more
arXiv (Cornell University)A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional\n Neural Networks for Sentence Classification
897 Citations2015Ye Zhang, Byron Wallace
Parsing with Compositional Vector Grammars
832 Citations2013Richard Socher, John Bauer +2 more
A Compositional Vector Grammar (CVG), which combines PCFGs with a syntactically untied recursive neural network that learns syntactico-semantic, compositional vector representations and improves performance on the types of ambiguities that require semantic information such as PP attachments.
Sentiment Analysis using Support Vector Machines with Diverse Information Sources
693 Citations2004Tony Mullen, Nigel Collier
Experiments on movie review data from the Internet Movie Database demonstrate that hybrid SVMs which combine unigram-style feature-based SVMs with those based on real-valued favorability measures obtain superior performance, producing the best results yet published using this data.
Convolutional-Recursive Deep Learning for 3D Object Classification
523 Citations2012Richard Socher, Brody Huval +3 more
This work introduces a model based on a combination of convolutional and recursive neural networks (CNN and RNN) for learning features and classifying RGB-D images, which obtains state of the art performance on a standardRGB-D object dataset while being more accurate and faster during training and testing than comparable architectures such as two-layer CNNs.
Comparative experiments on sentiment classification for online product reviews
318 Citations2006Hang Cui, Vibhu O. Mittal +1 more
A series of experiments with different machine learning algorithms are discussed in order to experimentally evaluate various trade-offs, using approximately 100K product reviews from the web.
Joint Language and Translation Modeling with Recurrent Neural Networks
259 Citations2013Michael Auli, Michel Galley +2 more
This work presents a joint language and translation model based on a recurrent neural network which predicts target words based on an unbounded history of both source and target words which shows competitive accuracy compared to the traditional channel model features.
Automatic identification of pro and con reasons in online reviews
225 Citations2006Soo-Min Kim, Eduard Hovy
A maximum entropy model is trained on the resulting labeled set to subsequently extract pros and cons from online review sites that do not explicitly provide them, and the resulting system identifies pros andcons with 66% precision and 76% recall.
A Recursive Recurrent Neural Network for Statistical Machine Translation
161 Citations2014Shujie Liu, Nan Yang +2 more
A novel recursive recurrent neural network (R 2 NN) is proposed to model the end-to-end decoding process for statistical machine translation and can outperform the state of theart baseline by about 1.5 points in BLEU.
International Joint Conference on Natural Language ProcessingAutomatic Detection of Opinion Bearing Words and Sentences
155 Citations2005Soo-Min Kim, Eduard Hovy
This work describes a sentence-level opinion detection system and introduces an effective method for obtaining opinion-bearing and nonopinion-bearing words that can be used effectively to identify opinion- bearing sentences.
Discriminative Neural Sentence Modeling by Tree-Based Convolution
84 Citations2015Lili Mou, Hao Peng +4 more
This paper proposes a tree-based convolutional neural network (TBCNN) for discriminative sentence modeling that outperforms previous state-of-the-art results, including existing neural networks and dedicated feature/rule engineering.
Compositional Distributional Semantics with Long Short Term Memory
69 Citations2015Phong Ba Le, Willem Zuidema
An extension of the recursive neural network that makes use of a variant of the long short-term memory architecture that provides a solution to the vanishing gradient problem and allows the network to capture long range dependencies.
arXiv (Cornell University)Long Short-Term Memory Over Tree Structures
68 Citations2015Xiaodan Zhu, Parinaz Sobhani +1 more
This paper proposes to extend chain-structured long short-term memory to tree structures, in which a memory cell can reflect the history memories of multiple child cells or multiple descendant cells in a recursive process, and calls the model S-LSTM, which provides a principled way of considering long-distance interaction over hierarchies.