Self-Adaptive Hierarchical Sentence Model
arXiv (Cornell University)Published 20 April 2015Open access
Han Zhao, Zhengdong Lu, Pascal Poupart
Citations52
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
The ability to accurately model a sentence at varying stages (e.g., word-phrase-sentence) plays a central role in natural language processing. As an effort towards this goal we propose a self-adaptive hierarchical sentence model (AdaSent). AdaSent effectively forms a hierarchy of representations from words to phrases and then to sentences through recursive gated local composition of adjacent segments. We design a competitive mechanism (through gating networks) to allow the representations of the same sentence to be engaged in a particular learning task (e.g., classification), therefore effectively mitigating the gradient vanishing problem persistent in other recursive models. Both qualitative and quantitative analysis shows that AdaSent can automatically form and select the representations suitable for the task at hand during training, yielding superior classification performance over competitor models on 5 benchmark data sets.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
arXiv (Cornell University)Distributed Representations of Words and Phrases and their Compositionality
18,086 Citations2013Tomáš Mikolov, Ilya Sutskever +3 more
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
Convolutional Neural Networks for Sentence Classification
13,798 Citations2014Yoon Kim
The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification, and are proposed to allow for the use of both task-specific and static vectors.
International Conference on Machine LearningRectified Linear Units Improve Restricted Boltzmann Machines
13,244 Citations2010Vinod Nair, Geoffrey E. Hinton
Restricted Boltzmann machines were developed using binary stochastic hidden units that learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset.
Cognitive ScienceFinding Structure in Time
10,841 Citations1990Jeffrey L. Elman
A proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory and suggests a method for representing lexical categories and the type/token distinction is developed.
IEEE Transactions on Signal ProcessingBidirectional recurrent neural networks
10,022 Citations1997Mike Schuster, Kuldip K. Paliwal
It is shown how the proposed bidirectional structure can be easily modified to allow efficient estimation of the conditional posterior probability of complete symbol sequences without making any explicit assumption about the shape of the distribution.
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
8,612 Citations2011John C. Duchi, Elad Hazan +1 more
IEEE Transactions on Neural NetworksLearning long-term dependencies with gradient descent is difficult
8,461 Citations1994Yoshua Bengio, P. Simard +1 more
This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods.
Mining and summarizing customer reviews
7,714 Citations2004Minqing Hu, Bing Liu
This research aims to mine and to summarize all the customer reviews of a product, and proposes several novel techniques to perform these tasks.
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
6,774 Citations2013Richard Socher, Alex Perelygin +5 more
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
6,616 Citations2014Kyunghyun Cho, Bart van Merriënboer +2 more
It is shown that the neural machine translation performs relatively well on short sentences without unknown words, but its performance degrades rapidly as the length of the sentence and the number of unknown words increase.
Neural ComputationLearning to Forget: Continual Prediction with LSTM
5,419 Citations2000Felix A. Gers, Jürgen Schmidhuber +1 more
This work identifies a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset, and proposes a novel, adaptive forget gate that enables an LSTm cell to learn to reset itself at appropriate times, thus releasing internal resources.
Recurrent neural network based language model
5,349 Citations2010Tomáš Mikolov, Martin Karafiát +3 more
Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model.
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
arXiv (Cornell University)Distributed Representations of Sentences and Documents
5,128 Citations2014Quoc V. Le, Tomáš Mikolov
Paragraph Vector is an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents, and its construction gives the algorithm the potential to overcome the weaknesses of bag-of-words models.
arXiv (Cornell University)On the difficulty of training Recurrent Neural Networks
3,778 Citations2012Razvan Pascanu, Tomáš Mikolov +1 more
This paper proposes a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem and validates empirically the hypothesis and proposed solutions.
A Convolutional Neural Network for Modelling Sentences
3,561 Citations2014Nal Kalchbrenner, Edward Grefenstette +1 more
A convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) is described that is adopted for the semantic modelling of sentences and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations.
A sentimental education
3,343 Citations2004Bo Pang, Lillian Lee
A novel machine-learning method is proposed that applies text-categorization techniques to just the subjective portions of the document, which greatly facilitates incorporation of cross-sentence contextual constraints.
Proceedings of the AAAI Conference on Artificial IntelligenceRecurrent Convolutional Neural Networks for Text Classification
2,306 Citations2015Siwei Lai, Liheng Xu +2 more
A recurrent convolutional neural network is introduced for text classification without human-designed features to capture contextual information as far as possible when learning word representations, which may introduce considerably less noise compared to traditional window-based neural networks.
Seeing stars
2,127 Citations2005Bo Pang, Lillian Lee
A meta-algorithm is applied, based on a metric labeling formulation of the rating-inference problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels.
Computers and the HumanitiesAnnotating Expressions of Opinions and Emotions in Language
1,770 Citations2005Janyce Wiebe, Theresa Wilson +1 more
The manual annotation process and the results of an inter-annotator agreement study on a 10,000-sentence corpus of articles drawn from the world press are presented.
Extensions of recurrent neural network language model
1,596 Citations2011Tomáš Mikolov, Stefan Kombrink +3 more
Several modifications of the original recurrent neural network language model are presented, showing approaches that lead to more than 15 times speedup for both training and testing phases and possibilities how to reduce the amount of parameters in the model.
Semantic Compositionality through Recursive Matrix-Vector Spaces
1,297 Citations2012Richard Socher, Brody Huval +2 more
A recursive neural network model that learns compositional vector representations for phrases and sentences of arbitrary syntactic type and length and can learn the meaning of operators in propositional logic and natural language is introduced.
Learning question classifiers
1,276 Citations2002Xin Li, Dan Roth
A hierarchical classifier is learned that is guided by a layered semantic hierarchy of answer types, and eventually classifies questions into fine-grained classes.
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
1,198 Citations2011Richard Socher, Jeffrey Pennington +3 more
A novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions that outperform other state-of-the-art approaches on commonly used datasets, without using any pre-defined sentiment lexica or polarity shifting rules.
arXiv (Cornell University)Convolutional Neural Network Architectures for Matching Natural Language Sentences
970 Citations2015Baotian Hu, Zhengdong Lu +2 more
Convolutional neural network models for matching two sentences are proposed, by adapting the convolutional strategy in vision and speech and nicely represent the hierarchical structures of sentences with their layer-by-layer composition and pooling.
Baselines and Bigrams: Simple, Good Sentiment and Topic Classification
967 Citations2012Sida Wang, Christopher D. Manning
It is shown that the inclusion of word bigram features gives consistent gains on sentiment analysis tasks, and a simple but novel SVM variant using NB log-count ratios as feature values consistently performs well across tasks and datasets.
Proceedings of International Conference on Neural Networks (ICNN'96)Learning task-dependent distributed representations by backpropagation through structure
592 Citations2002Christoph Goller, Andreas Küchler
neural information processing systemsDeep Recursive Neural Networks for Compositionality in Language
259 Citations2014Ozan İrsoy, Claire Cardie
The results show that deep RNNs outperform associated shallow counterparts that employ the same number of parameters and outperforms previous baselines on the sentiment analysis task, including a multiplicative RNN variant as well as the recently introduced paragraph vectors.
Artificial Intelligence ReviewFrom symbolic to sub-symbolic information in question classification
156 Citations2010João Silva, Luísa Coheur +2 more
This paper presents and evaluates a rule-based question classifier that partially founds its performance in the detection of the question headword and in its mapping into the target category through the use of WordNet, and uses the rule-base classifier as a features’ provider of a machine learning-basedquestion classifier.