Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection
Published 12 December 2011
Richard Socher, Eric Huang, Jeffrey Pennin, Christopher D. Manning, Andrew Y. Ng
Citations810
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work introduces a method for paraphrase detection based on recursive autoencoders (RAE) and unsupervised RAEs based on a novel unfolding objective and learns feature vectors for phrases in syntactic trees to measure word- and phrase-wise similarity between two sentences.
Abstract
Paraphrase detection is the task of examining two sentences and determining whether they have the same meaning. In order to obtain high accuracy on this task, thorough syntactic and semantic analysis of the two statements is needed. We introduce a method for paraphrase detection based on recursive autoencoders (RAE). Our unsupervised RAEs are based on a novel unfolding objective and learn feature vectors for phrases in syntactic trees. These features are used to measure the word- and phrase-wise similarity between two sentences. Since sentences may be of arbitrary length, the resulting matrix of similarity measures is of variable size. We introduce a novel dynamic pooling layer which computes a fixed-sized representation from the variable-sized matrices. The pooled representation is then used as input to a classifier. Our method outperforms other state-of-the-art ap-proaches on the challenging MSRP paraphrase corpus. 1
Keywords
Computer Science
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
Curriculum learning
4,978 Citations2009Yoshua Bengio, Jérôme Louradour +2 more
It is hypothesized that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Accurate unlexicalized parsing
3,055 Citations2003Dan Klein, Christopher D. Manning
It is demonstrated that an unlexicalized PCFG can parse much more accurately than previously shown, by making use of simple, linguistically motivated state splits, which break down false independence assumptions latent in a vanilla treebank grammar.
Word Representations: A Simple and General Method for Semi-Supervised Learning
1,944 Citations2010Joseph Turian, Lev-Arie Ratinov +1 more
This work evaluates Brown clusters, Collobert and Weston (2008) embeddings, and HLBL (Mnih & Hinton, 2009) embeds of words on both NER and chunking, and finds that each of the three word representations improves the accuracy of these baselines.
Applied Physics Letters10.1162/153244303322533223
1,771 Citations2000
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences.
Machine LearningDistributed representations, simple recurrent networks, and grammatical structure
1,381 Citations1991Jeffrey L. Elman
Parsing Natural Scenes and Natural Language with Recursive Neural Networks
1,209 Citations2011Richard Socher, Cliff Chiung-Yu Lin +2 more
A max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences is introduced.
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
1,198 Citations2011Richard Socher, Jeffrey Pennington +3 more
A novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions that outperform other state-of-the-art approaches on commonly used datasets, without using any pre-defined sentiment lexica or polarity shifting rules.
Corpus-based and knowledge-based measures of text semantic similarity
1,189 Citations2006Rada Mihalcea, Courtney D. Corley +1 more
This paper shows that the semantic similarity method out-performs methods based on simple lexical matching, resulting in up to 13% error rate reduction with respect to the traditional vector-based similarity metric.
Journal of Machine Learning ResearchExploring Strategies for Training Deep Neural Networks
1,114 Citations2009Hugo Larochelle, Yoshua Bengio +2 more
Artificial IntelligenceRecursive distributed representations
866 Citations1990Jordan Pollack
This paper presents a connectionist architecture which automatically develops compact distributed representations for variable-sized recursive data structures, as well as efficient accessing mechanisms for them.
Unsupervised construction of large paraphrase corpora
740 Citations2004Bill Dolan, Chris Quirk +1 more
Investigation of unsupervised techniques for acquiring monolingual sentence-level paraphrases from a corpus of temporally and topically clustered news articles collected from thousands of web-based news sources shows that edit distance data is cleaner and more easily-aligned than the heuristic data.
Proceedings of International Conference on Neural Networks (ICNN'96)Learning task-dependent distributed representations by backpropagation through structure
592 Citations2002Christoph Goller, Andreas Küchler
Learning to paraphrase
432 Citations2003Regina Barzilay, Lillian Lee
This work applies multiple-sequence alignment to sentences gathered from unannotated comparable corpora: it learns a set of paraphrasing patterns represented by word lattice pairs and automatically determines how to apply these patterns to rewrite new sentences.
Paraphrase identification as probabilistic quasi-synchronous recognition
215 Citations2009Dipanjan Das, Noah A. Smith
A generative model is employed that generates a paraphrase of a given sentence, and probabilistic inference is used to reason about whether two sentences share the paraphrase relationship.
Syntactic constraints on paraphrases extracted from parallel corpora
183 Citations2008Chris Callison-Burch
This work improves the quality of paraphrases extracted from parallel corpora by requiring that phrases and their paraphrasing be the same syntactic type and altering the phrase extraction algorithm to extract phrase labels alongside bilingual phrase pairs.
Using Dependency-Based Features to Take the 'Para-farce' out of Paraphrase
143 Citations2006Stephen Wan, Mark Dras +2 more
A machine learning approach is proposed to be used to filter out inconsistent novel sentences, or False Paraphrases, using the Microsoft Research Paraphrase corpus and investigating whether features based on syntactic dependencies can aid in this task.
Paraphrase recognition via dissimilarity significance classification
104 Citations2006Long Qiu, Min‐Yen Kan +1 more
Experimental results show that while being accurate at discerning non-paraphrasing dissimilarities, the implemented system is able to achieve higher paraphrase recall (93%), at an overall performance comparable to the alternatives.
Research portal (Tilburg University)Explorations in Sentence Fusion
90 Citations2005Erwin Marsi, Emiel Krahmer
Lecture notes in computer scienceParaphrase Identification on the Basis of Supervised Machine Learning Techniques
83 Citations2006Zornitsa Kozareva, Andrés Montoyo
The objective to increase the final performance of the system is to scrutinize the influence of the combination of lexical and semantic information, as well as techniques for classifier combination.
Paraphrase Identification with Lexico-Syntactic Graph Subsumption.
73 Citations2008Vasile Rus, Philip M. McCarthy +3 more
The paper presents a new approach to the problem of paraphrase identification that extends a previously proposed method for the task of textual entailment and offers significantly better results than two baselines.
Paraphrase Identification by Text Canonicalization
62 Citations2005Yitao Zhang, Jon Patrick
An approach to sentence-level paraphrase identification by text canon-icalization by text canon-icalization is proposed, which suggests that the MSR Para-phrase Corpus might not be a rich source for learning paraphrasing patterns.
Amsterdam studies in the theory and history of linguistic science. Series 4, Current issues in linguistic theorySemantic similarity of short texts
49 Citations2009Aminul Islam, Diana Inkpen
This paper presents a method for measuring the semantic similarity of texts using a corpus based measure of semantic word similarity and a normalized and modified versions of the Longest Common Subsequence string matching algorithm.
Neural NetworksLinear recursive distributed representations
21 Citations2005Thomas Voegtlin, Peter Ford Dominey
A modification of Pollack's Recursive Auto-Associative Memory, that is trained with Oja's rule, in which it generalizes PCA to tree-structured data and displays developmental effects similar to those observed in children when learning to generalize about the argument structure of verbs.