Deep Unordered Composition Rivals Syntactic Methods for Text Classification
Published 1 January 2015Open access
Mohit Iyyer, Varun Manjunatha, Jordan Boyd‐Graber, Hal Daumé
Citations741
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work presents a simple deep neural network that competes with and, in some cases, outperforms such models on sentiment analysis and factoid question answering tasks while taking only a fraction of the training time.
Abstract
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, Hal Daumé III. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.
Keywords
Computer Science
Dropout: a simple way to prevent neural networks from overfitting
34,279 Citations2014Nitish Srivastava, Geoffrey E. Hinton +3 more
It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
24,447 Citations2014Kyunghyun Cho, Bart van Merriënboer +5 more
Qualitatively, the proposed RNN Encoder‐Decoder model learns a semantically and syntactically meaningful representation of linguistic phrases.
Convolutional Neural Networks for Sentence Classification
13,798 Citations2014Yoon Kim
The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification, and are proposed to allow for the use of both task-specific and static vectors.
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
IEEE Transactions on Pattern Analysis and Machine IntelligenceRepresentation Learning: A Review and New Perspectives
13,002 Citations2013Yoshua Bengio, Aaron Courville +1 more
Recent work in the area of unsupervised feature learning and deep learning is reviewed, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks.
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
8,612 Citations2011John C. Duchi, Elad Hazan +1 more
Mining and summarizing customer reviews
7,714 Citations2004Minqing Hu, Bing Liu
This research aims to mine and to summarize all the customer reviews of a product, and proposes several novel techniques to perform these tasks.
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
6,774 Citations2013Richard Socher, Alex Perelygin +5 more
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
arXiv (Cornell University)Improving neural networks by preventing co-adaptation of feature detectors
6,631 Citations2012Geoffrey E. Hinton, Nitish Srivastava +3 more
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
arXiv (Cornell University)Distributed Representations of Sentences and Documents
5,128 Citations2014Quoc V. Le, Tomáš Mikolov
Paragraph Vector is an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents, and its construction gives the algorithm the potential to overcome the weaknesses of bag-of-words models.
A Convolutional Neural Network for Modelling Sentences
3,561 Citations2014Nal Kalchbrenner, Edward Grefenstette +1 more
A convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) is described that is adopted for the semantic modelling of sentences and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations.
Learning Word Vectors for Sentiment Analysis
3,299 Citations2011Andrew L. Maas, Raymond E. Daly +4 more
This work presents a model that uses a mix of unsupervised and supervised techniques to learn word vectors capturing semantic term--document information as well as rich sentiment content, and finds it out-performs several previously introduced methods for sentiment classification.
Journal of Machine Learning ResearchA neural probabilistic language model
2,663 Citations2003BengioYoshua, DucharmeRéjean +2 more
This work reports on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach very significantly improves on a state-of-the-art trigram model.
Seeing stars
2,127 Citations2005Bo Pang, Lillian Lee
A meta-algorithm is applied, based on a metric labeling formulation of the rating-inference problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels.
A Fast and Accurate Dependency Parser using Neural Networks
1,879 Citations2014Danqi Chen, Christopher D. Manning
This work proposes a novel way of learning a neural network classifier for use in a greedy, transition-based dependency parser that can work very fast, while achieving an about 2% improvement in unlabeled and labeled attachment scores on both English and Chinese datasets.
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
1,198 Citations2011Richard Socher, Jeffrey Pennington +3 more
A novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions that outperform other state-of-the-art approaches on commonly used datasets, without using any pre-defined sentiment lexica or polarity shifting rules.
Baselines and Bigrams: Simple, Good Sentiment and Topic Classification
967 Citations2012Sida Wang, Christopher D. Manning
It is shown that the inclusion of word bigram features gives consistent gains on sentiment analysis tasks, and a simple but novel SVM variant using NB log-count ratios as feature values consistently performs well across tasks and datasets.
Parsing with Compositional Vector Grammars
832 Citations2013Richard Socher, John Bauer +2 more
A Compositional Vector Grammar (CVG), which combines PCFGs with a syntactically untied recursive neural network that learns syntactico-semantic, compositional vector representations and improves performance on the types of ambiguities that require semantic information such as PP attachments.
Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection
810 Citations2011Richard Socher, Eric Huang +3 more
This work introduces a method for paraphrase detection based on recursive autoencoders (RAE) and unsupervised RAEs based on a novel unfolding objective and learns feature vectors for phrases in syntactic trees to measure word- and phrase-wise similarity between two sentences.
arXiv (Cornell University)Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
725 Citations2005Bo Pang, Lillian Lee
Edinburgh Research Explorer (University of Edinburgh)Vector-based Models of Semantic Composition
666 Citations2008Jeff Mitchell, Mirella Lapata
Under this framework, a wide range of composition models are introduced which are evaluated empirically on a sentence similarity task and demonstrate that the multiplicative models are superior to the additive alternatives when compared against human judgments.
Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space
485 Citations2010Marco Baroni, Roberto Zamparelli
This work proposes an approach to adjective-noun composition (AN) for corpus-based distributional semantics that represents nouns as vectors and adjectives as data-induced (linear) functions over nominal vectors, and shows that the model significantly outperforms the rivals on the task of reconstructing AN vectors not seen in training.
A structured vector space model for word meaning in context
362 Citations2008Katrin Erk, Sebastian Padó
A novel structured vector space model is presented that makes it possible to integrate syntax into the computation of word meaning in context and performs at and above the state of the art for modeling the contextual adequacy of paraphrases.
A Neural Network for Factoid Question Answering over Paragraphs
323 Citations2014Mohit Iyyer, Jordan Boyd‐Graber +3 more
This work introduces a recursive neural network model, qanta, that can reason over question text input by modeling textual compositionality and applies it to a dataset of questions from a trivia competition called quiz bowl.
Political Ideology Detection Using Recursive Neural Networks
282 Citations2014Mohit Iyyer, Peter K. Enns +2 more
A RNN framework is applied to the task of identifying the political position evinced by a sentence to show the importance of modeling subsentential elements and outperforms existing models on a newly annotated dataset and an existing dataset.
neural information processing systemsDeep Recursive Neural Networks for Compositionality in Language
259 Citations2014Ozan İrsoy, Claire Cardie
The results show that deep RNNs outperform associated shallow counterparts that employ the same number of parameters and outperforms previous baselines on the sentiment analysis task, including a multiplicative RNN variant as well as the recently introduced paragraph vectors.
arXiv (Cornell University)Experimental Support for a Categorical Compositional Distributional Model of Meaning
225 Citations2011Edward Grefenstette, Mehrnoosh Sadrzadeh
The abstract categorical model of Coecke et al. (2010) is implemented using data from the BNC and evaluated, with general improvement in results with increase in syntactic complexity showcasing the compositional power of the model.
arXiv (Cornell University)Recurrent Convolutional Neural Networks for Discourse Compositionality
192 Citations2013Nal Kalchbrenner, Phil Blunsom
The discourse model coupled to the sentence model obtains state of the art performance on a dialogue act classification experiment and is able to capture both the sequentiality of sentences and the interaction between different speakers.
A Recursive Recurrent Neural Network for Statistical Machine Translation
161 Citations2014Shujie Liu, Nan Yang +2 more
A novel recursive recurrent neural network (R 2 NN) is proposed to model the end-to-end decoding process for statistical machine translation and can outperform the state of theart baseline by about 1.5 points in BLEU.
arXiv (Cornell University)Mathematical Foundations for a Compositional Distributional Model of\n Meaning
148 Citations2010Bob Coecke, Mehrnoosh Sadrzadeh +1 more
Recurrent Convolutional Neural Networks for Discourse Compositionality
76 Citations2013Nal Kalchbrenner, Phil Blunsom
Prior Disambiguation of Word Tensors for Constructing Sentence Vectors
75 Citations2013Dimitri Kartsaklis, Mehrnoosh Sadrzadeh
This paper proposes disambiguation algorithms for a number of tensor-based models, and tests the effectiveness of these algorithms on a variety of tasks, showing that disambIGuation can provide better compositional representation even for the case of tensors.
A Neural Network Approach to Selectional Preference Acquisition
55 Citations2014Tim Van de Cruys
A neural network model that learns to discriminate between felicitous and infelicitous arguments for a particular predicate is proposed, which is entirely unsupervised ‐ preferences are learned from unannotated corpus data.
arXiv (Cornell University)Training Restricted Boltzmann Machines on Word Observations
50 Citations2012George E. Dahl, Hugo Larochelle +1 more
The success of this approach is demonstrated by training RBMs on hundreds of millions of word n-grams using larger vocabularies than previously feasible with RBMs and by using the learned features to improve performance on chunking and sentiment classification tasks, achieving state-of-the-art results on the latter.
Grammatical structures for word-level sentiment detection
41 Citations2012Asad Sayeed, Jordan Boyd‐Graber +2 more
This work proposes a suffix-tree data structure to represent syntactic relationships between opinion targets and words in a sentence that are opinion-bearing and shows that a factor graph derived from this data structure acquires these relationships with a small number of word-level features.
Empirical Methods in Natural Language ProcessingBesting the Quiz Master: Crowdsourcing Incremental Classification Games
35 Citations2012Jordan Boyd‐Graber, Brianna Satinoff +2 more
This work introduces a setting where humans engage in classification with incrementally revealed features: the collegiate trivia circuit and improves the performance of a state-of-the art classifier by observing humans' classification process.
SimCompass: Using Deep Learning Word Embeddings to Assess Cross-level Similarity
33 Citations2014Carmen Banea, Di Chen +3 more
This article presents the team’s participating system at SemEval-2014 Task 3.0 using a meta-learning framework, which enabled the team to reach the highest overall performance among all competing systems.
arXiv (Cornell University)"Not not bad" is not "bad": A distributional account of negation
21 Citations2013Karl Moritz Hermann, Edward Grefenstette +1 more
This paper proposes a tripartite formulation for a continuous vector space representation of semantics and uses this representation to develop a formal compositional notion of negation within such models as distributional models of compositional semantics.
arXiv (Cornell University)Feature Weight Tuning for Recursive Neural Networks
13 Citations2014Jiwei Li
This paper addresses how a recursive neural network model can automatically leave out useless information and emphasize important evidence, in other words, to perform "weight tuning" for higher-level representation acquisition.