The Role of Syntax in Vector Space Models of Compositional Semantics
Oxford University Research Archive (ORA) (University of Oxford)Published 1 August 2013Open access
Karl Moritz Hermann, Phil Blunsom
Citations158
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This model leverages the CCG combinatory operators to guide a non-linear transformation of meaning within a sentence and is used to learn high dimensional embeddings for sentences and evaluate them in a range of tasks, demonstrating that the incorporation of syntax allows a concise model to learn representations that are both effective and general.
Abstract
Modelling the compositional process by which the meaning of an utterance arises from the meaning of its parts is a fundamental task of Natural Language Processing.In this paper we draw upon recent advances in the learning of vector space representations of sentential semantics and the transparent interface between syntax and semantics provided by Combinatory Categorial Grammar to introduce Combinatory Categorial Autoencoders.This model leverages the CCG combinatory operators to guide a non-linear transformation of meaning within a sentence.We use this model to learn high dimensional embeddings for sentences and evaluate them in a range of tasks, demonstrating that the incorporation of syntax allows a concise model to learn representations that are both effective and general.
Keywords
Computer Science
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
The MIT Press eBooksGreedy Layer-Wise Training of Deep Networks
4,704 Citations2007Yoshua Bengio, Pascal Lamblin +2 more
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.
LanguageSemantic Interpretation in Generative Grammar
4,650 Citations1975Robert Freidin, Ray Jackendoff
This book investigates a wide variety of semantic rules, stating them in considerable detail and extensively treating their consequences for the syntactic component of the grammar, and proposes radically new approaches to the so-called Crossover Principle, the control problem for complement subjects, parentheticals, and the interpretation of nonspecific noun phrases.
Recognizing contextual polarity in phrase-level sentiment analysis
3,379 Citations2005Theresa Wilson, Janyce Wiebe +1 more
A new approach to phrase-level sentiment analysis is presented that first determines whether an expression is neutral or polar and then disambiguates the polarity of the polar expressions.
Seeing stars
2,127 Citations2005Bo Pang, Lillian Lee
A meta-algorithm is applied, based on a metric labeling formulation of the rating-inference problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels.
Word Representations: A Simple and General Method for Semi-Supervised Learning
1,944 Citations2010Joseph Turian, Lev-Arie Ratinov +1 more
This work evaluates Brown clusters, Collobert and Weston (2008) embeddings, and HLBL (Mnih & Hinton, 2009) embeds of words on both NER and chunking, and finds that each of the three word representations improves the accuracy of these baselines.
Computers and the HumanitiesAnnotating Expressions of Opinions and Emotions in Language
1,770 Citations2005Janyce Wiebe, Theresa Wilson +1 more
The manual annotation process and the results of an inter-annotator agreement study on a 10,000-sentence corpus of articles drawn from the world press are presented.
Semantic Compositionality through Recursive Matrix-Vector Spaces
1,297 Citations2012Richard Socher, Brody Huval +2 more
A recursive neural network model that learns compositional vector representations for phrases and sentences of arbitrary syntactic type and length and can learn the meaning of operators in propositional logic and natural language is introduced.
Über Sinn und Bedeutung
1,287 Citations1892Gottlob Frege
Automatic word sense discrimination
1,286 Citations1998Hinrich Schütze
This paper presents context-group discrimination, a disambiguation algorithm based on clustering that demonstrates good performance of context- group discrimination for a sample of natural and artificial ambiguous words.
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
1,198 Citations2011Richard Socher, Jeffrey Pennington +3 more
A novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions that outperform other state-of-the-art approaches on commonly used datasets, without using any pre-defined sentiment lexica or polarity shifting rules.
Distributional clustering of English words
994 Citations1993Fernando Pereira, Naftali Tishby +1 more
Cognitive ScienceComposition in Distributional Models of Semantics
981 Citations2010Jeff Mitchell, Mirella Lapata
This article proposes a framework for representing the meaning of word combinations in vector space in terms of additive and multiplicative functions, and introduces a wide range of composition models that are evaluated empirically on a phrase similarity task.
Baselines and Bigrams: Simple, Good Sentiment and Topic Classification
967 Citations2012Sida Wang, Christopher D. Manning
It is shown that the inclusion of word bigram features gives consistent gains on sentiment analysis tasks, and a simple but novel SVM variant using NB log-count ratios as feature values consistently performs well across tasks and datasets.
Artificial IntelligenceRecursive distributed representations
866 Citations1990Jordan Pollack
This paper presents a connectionist architecture which automatically develops compact distributed representations for variable-sized recursive data structures, as well as efficient accessing mechanisms for them.
Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection
810 Citations2011Richard Socher, Eric Huang +3 more
This work introduces a method for paraphrase detection based on recursive autoencoders (RAE) and unsupervised RAEs based on a novel unfolding objective and learns feature vectors for phrases in syntactic trees to measure word- and phrase-wise similarity between two sentences.
Edinburgh Research Explorer (University of Edinburgh)Vector-based Models of Semantic Composition
666 Citations2008Jeff Mitchell, Mirella Lapata
Under this framework, a wide range of composition models are introduced which are evaluated empirically on a sentence similarity task and demonstrate that the multiplicative models are superior to the additive alternatives when compared against human judgments.
Proceedings of International Conference on Neural Networks (ICNN'96)Learning task-dependent distributed representations by backpropagation through structure
592 Citations2002Christoph Goller, Andreas Küchler
Convolutional-Recursive Deep Learning for 3D Object Classification
523 Citations2012Richard Socher, Brody Huval +3 more
This work introduces a model based on a combination of convolutional and recursive neural networks (CNN and RNN) for learning features and classifying RGB-D images, which obtains state of the art performance on a standardRGB-D object dataset while being more accurate and faster during training and testing than comparable architectures such as two-layer CNNs.
Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space
485 Citations2010Marco Baroni, Roberto Zamparelli
This work proposes an approach to adjective-noun composition (AN) for corpus-based distributional semantics that represents nouns as vectors and adjectives as data-induced (linear) functions over nominal vectors, and shows that the model significantly outperforms the rivals on the task of reconstructing AN vectors not seen in training.
Computational LinguisticsWide-Coverage Efficient Statistical Parsing with CCG and Log-Linear Models
409 Citations2007Stephen Clark, James Curran
This article describes a number of log-linear parsing models for an automatically extracted lexicalized grammar and develops a new model and efficient parsing algorithm which exploits all derivations, including CCG's nonstandard derivations.
Computational LinguisticsCCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank
387 Citations2007Julia Hockenmaier, Mark Steedman
This article presents an algorithm for translating the Penn Treebank into a corpus of Combinatory Categorial Grammar (CCG) derivations augmented with local and long-range word-word dependencies, and discusses the implications of the findings for the extraction of other linguistically expressive grammars from the Treebank, and for the design of future treebanks.
Dependency Tree-based Sentiment Classification using CRFs with Hidden Variables
336 Citations2010Tetsuji Nakagawa, Kentaro Inui +1 more
Experimental results of sentiment classification of Japanese and English subjective sentences using conditional random fields with hidden variables showed that the method performs better than other methods based on bag-of-features.
Combinatory Categorial Grammar
257 Citations2011Mark Steedman, Jason Baldridge
The paper reviews its use for the explanatory analysis of natural language grammar, particularly with respect to bounded and unbounded constructions, multiple dependencies, relativization, and coordination.
Linguistically motivated large-scale NLP with C&C and boxer
254 Citations2007James Curran, Stephen Clark +1 more
An NLP system which is based on syntactic and semantic formalisms from theoretical linguistics, and which is used to analyse the entire Gigaword corpus in less than 5 days using only 18 processors, represents a break-through in NLP technology.
A Comparison of Vector-based Representations for Semantic Composition
246 Citations2012William Blacoe, Mirella Lapata
Shallow approaches to modeling compositional meaning for phrases and sentences using distributional methods are found to be as good as more computationally intensive alternatives with regards to two particular tests: phrase similarity and paraphrase detection.
arXiv (Cornell University)Experimental Support for a Categorical Compositional Distributional Model of Meaning
225 Citations2011Edward Grefenstette, Mehrnoosh Sadrzadeh
The abstract categorical model of Coecke et al. (2010) is implemented using data from the BNC and evaluated, with general improvement in results with increase in syntactic complexity showcasing the compositional power of the model.
Automatic identification of non-compositional phrases
208 Citations1999Dekang Lin
This work presents a method for automatic identification of non-compositional expressions using their statistical properties in a text corpus based on the hypothesis that when a phrase is non-Compositional, its mutual information differs significantly from the mutual informations of phrases obtained by substituting one of the word in the phrase with a similar word.
Cognitive ScienceUnsupervised Discovery of Nonlinear Structure Using Contrastive Backpropagation
137 Citations2006Geoffrey E. Hinton, Simon Osindero +2 more
A way of modeling high-dimensional data vectors by using an unsupervised, nonlinear, multilayer neural network in which the activity of each neuron-like unit makes an additive contribution to a global energy score that indicates how surprised the network is by the data vector.
Developing a large semantically annotated corpus
115 Citations2012Valerio Basile, Johan Bos +2 more
It is argued that a bootstrapping approach comprising state-of-the-art NLP tools for parsing and semantic interpretation, in combination with a wiki-like interface for collaborative annotation of experts, and a game with a purpose for crowdsourcing, are the starting ingredients for fulfilling this enterprise.
Oxford University Research Archive (ORA) (University of Oxford)Multi-Step Regression Learning for Compositional Distributional Semantics
57 Citations2013Edward Grefenstette, Georgiana Dinu +3 more
It is argued in the analysis that the nature of this learning method also renders it suitable for solving more subtle problems compositional distributional models might face, and is found to outperform existing leading methods.