Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Published 1 January 2013Open access
Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng
Citations6,774
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
Abstract
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, Christopher Potts. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 2013.
Keywords
Computer Science
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
8,612 Citations2011John C. Duchi, Elad Hazan +1 more
now publishers, Inc. eBooksOpinion Mining and Sentiment Analysis
6,745 Citations2008Bo Pang, Lillian Lee
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
Accurate unlexicalized parsing
3,055 Citations2003Dan Klein, Christopher D. Manning
It is demonstrated that an unlexicalized PCFG can parse much more accurately than previously shown, by making use of simple, linguistically motivated state splits, which break down false independence assumptions latent in a vanilla treebank grammar.
Journal of Artificial Intelligence ResearchFrom Frequency to Meaning: Vector Space Models of Semantics
2,883 Citations2010Peter D. Turney, Patrick Pantel
The goal in this survey is to show the breadth of applications of VSMs for semantics, to provide a new perspective on VSMs, and to provide pointers into the literature for those who are less familiar with the field.
Seeing stars
2,127 Citations2005Bo Pang, Lillian Lee
A meta-algorithm is applied, based on a metric labeling formulation of the rating-inference problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels.
Semantic Compositionality through Recursive Matrix-Vector Spaces
1,297 Citations2012Richard Socher, Brody Huval +2 more
A recursive neural network model that learns compositional vector representations for phrases and sentences of arbitrary syntactic type and length and can learn the meaning of operators in propositional logic and natural language is introduced.
Parsing Natural Scenes and Natural Language with Recursive Neural Networks
1,209 Citations2011Richard Socher, Cliff Chiung-Yu Lin +2 more
A max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences is introduced.
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
1,198 Citations2011Richard Socher, Jeffrey Pennington +3 more
A novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions that outperform other state-of-the-art approaches on commonly used datasets, without using any pre-defined sentiment lexica or polarity shifting rules.
Improving Word Representations via Global Context and Multiple Word Prototypes
1,101 Citations2012Eric Huang, Richard Socher +2 more
A new neural network architecture is presented which learns word embeddings that better capture the semantics of words by incorporating both local and global document context, and accounts for homonymy and polysemy by learning multiple embedDings per word.
Cognitive ScienceComposition in Distributional Models of Semantics
981 Citations2010Jeff Mitchell, Mirella Lapata
This article proposes a framework for representing the meaning of word combinations in vector space in terms of additive and multiplicative functions, and introduces a wide range of composition models that are evaluated empirically on a phrase similarity task.
Artificial IntelligenceRecursive distributed representations
866 Citations1990Jordan Pollack
This paper presents a connectionist architecture which automatically develops compact distributed representations for variable-sized recursive data structures, as well as efficient accessing mechanisms for them.
arXiv (Cornell University)Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
791 Citations2012Luke Zettlemoyer, Michael J. Collins
Computational LinguisticsDependency-Based Construction of Semantic Space Models
701 Citations2007Sebastian Padó, Mirella Lapata
This article presents a novel framework for constructing semantic spaces that takes syntactic relations into account, and introduces a formalization for this class of models, which allows linguistic knowledge to guide the construction process.
Computational LinguisticsDistributional Memory: A General Framework for Corpus-Based Semantics
662 Citations2010Marco Baroni, Alessandro Lenci
The Distributional Memory approach is shown to be tenable despite the constraints imposed by its multi-purpose nature, and performs competitively against task-specific algorithms recently reported in the literature for the same tasks, and against several state-of-the-art methods.
IEEE Transactions on Neural NetworksHolographic reduced representations
660 Citations1995Tony Plate
This paper describes a method for representing more complex compositional structure in distributed representations that uses circular convolution to associate items, which are represented by vectors.
Proceedings of International Conference on Neural Networks (ICNN'96)Learning task-dependent distributed representations by backpropagation through structure
592 Citations2002Christoph Goller, Andreas Küchler
A System for Real-time Twitter Sentiment Analysis of 2012 U.S. Presidential Election Cycle
551 Citations2012Hao Wang, Doğan Can +3 more
A system for real-time analysis of public sentiment toward presidential candidates in the 2012 U.S. election as expressed on Twitter, a micro-blogging service, offers a new and timely perspective on the dynamics of the electoral process and public opinion.
The information retrieval seriesContextual Valence Shifters
518 Citations2006Livia Polanyi, Annie Zaenen
It is argued that the current work in this area that concentrates mainly on the negative or positive attitude communicated by individual terms is incomplete and often gives the wrong results when implemented directly and proposes a simple, “proof of concept” implementation for some contextual shifters.
HAL (Le Centre pour la Communication Scientifique Directe)A latent factor model for highly multi-relational data
366 Citations2012Rodolphe Jenatton, Nicolas Le Roux +2 more
This paper proposes a method for modeling large multi-relational datasets, with possibly thousands of relations, based on a bilinear structure, which captures various orders of interaction of the data and also shares sparse latent factors across different relations.
A structured vector space model for word meaning in context
362 Citations2008Katrin Erk, Sebastian Padó
A novel structured vector space model is presented that makes it possible to integrate syntax into the computation of word meaning in context and performs at and above the state of the art for modeling the contextual adequacy of paraphrases.
Dependency Tree-based Sentiment Classification using CRFs with Hidden Variables
336 Citations2010Tetsuji Nakagawa, Kentaro Inui +1 more
Experimental results of sentiment classification of Japanese and English subjective sentences using conditional random fields with hidden variables showed that the method performs better than other methods based on bag-of-features.
Multiple Aspect Ranking Using the Good Grief Algorithm
316 Citations2007Benjamin Snyder, Regina Barzilay
An algorithm is presented that jointly learns ranking models for individual aspects by modeling the dependencies between assigned ranks, and it is proved that the agreementbased joint model is more expressive than individual ranking models.
Artificial IntelligenceMapping part-whole hierarchies into connectionist networks
311 Citations1990Geoffrey E. Hinton
Three different ways of mapping part-whole hierarchies into connectionist networks are described, suggesting that neural networks have two quite different methods for performing inference.
Journal of SemanticsMinimizers, Maximizers and the Rhetoric of Scalar Reasoning
251 Citations2001Michael Israel
The lexicalization patterns of polarity items are examined and the exceptional behaviour of these forms is shown to be a direct function of the participant roles they denote, reflecting the fact that different roles license different kinds of scalar inferences depending on how they contribute to the likelihood of an expressed proposition.
DSpace@MIT (Massachusetts Institute of Technology)Modelling Relational Data using Bayesian Clustered Tensor Factorization
241 Citations2009Ilya Sutskever, Joshua B. Tenenbaum +1 more
The Bayesian Clustered Tensor Factorization (BCTF) model is introduced, which embeds a factorized representation of relations in a nonparametric Bayesian clustering framework that is fully Bayesian but scales well to large data sets.
arXiv (Cornell University)Experimental Support for a Categorical Compositional Distributional Model of Meaning
225 Citations2011Edward Grefenstette, Mehrnoosh Sadrzadeh
The abstract categorical model of Coecke et al. (2010) is implemented using data from the BNC and evaluated, with general improvement in results with increase in syntactic complexity showcasing the compositional power of the model.
Factored 3-Way Restricted Boltzmann Machines For Modeling Natural Images
194 Citations2010Marc’Aurelio Ranzato, Alex Krizhevsky +1 more
A factored 3-way RBM is proposed that uses the states of its hidden units to represent abnormalities in the local covariance structure of an image to provide a probabilistic framework for the widely used simple/complex cell architecture.
Compositional Matrix-Space Models for Sentiment Analysis
164 Citations2011Ainur Yessenalina, Claire Cardie
This paper presents the first such algorithm for learning a matrix-space model for semantic composition, and its experimental results show statistically significant improvements in performance over a bag-of-words model.
Cineca Institutional Research Information System (Tor Vergata University)Estimating Linear Models for Compositional Distributional Semantics
126 Citations2010Fabio Massimo Zanzotto, Ioannis Korkontzelos +2 more
This paper proposes a novel approach to estimate parameters for a class of compositional distributional models: the additive models, and demonstrates that this approach outperforms existing methods for determining a good model for compositional distributionsal semantics.
Oxford University Research Archive (ORA) (University of Oxford)Combining Symbolic and Distributional Models of Meaning
119 Citations2007Stephen Clark, Stephen Pulman
A method is to be adapted from the Cognitive Science literature, in which symbolic and connectionist representations are combined using tensor products, to adapt a method for language processing.
Information, relevance, and social decisionmaking: some principles and results of decision-theoretic semantics
114 Citations1999Arthur Merin
An approach to meaning is submitted, with a focus on broadly nontruth-conditional aspects of natural language, and a key theoretical notion, distinguishable from, but intelligibly related to, information is the positive or negative relevance of a proposition or sentence to another.
Semantic Vector Products: Some Initial Investigations
96 Citations2008Dominic Widdows
This paper explores some of the opportunities for vector product operations to model compositional phenomena in natural language, and gathers a list of existing operators, and a lists of typical composition operations in naturallanguage, and describes two small experiments that begin to investigate the use of certain vector operators to model certain language phenomena.
Compositional Matrix-Space Models of Language
53 Citations2010Sebastian Rudolph, Eugenie Giesbrecht
CMSMs, a novel type of generic compositional models for syntactic and semantic aspects of natural language, based on matrix multiplication is proposed and it is shown that it is able to cover and combine various common compositional NLP approaches.
Multi-entity Sentiment Scoring
45 Citations2009Karo Moilanen, Stephen Pulman
A compositional framework for modelling entity-level sentiment (sub)contexts is presented, and it is demonstrated how holistic multi-entity polarity scoring emerges as a by-product of compositional sentiment parsing.
Large vocabulary speech recognition using deep tensor neural networks
23 Citations2012Dong Yu, Li Deng +1 more
Evaluation on 30hr Switchboard task indicates that DTNNs can outperform DNNs with similar number of parameters with 5% relative word error reduction, and is extended to deep tensor neural networks (DTNNs) in which one or more layers are double-projection and tensor layers.