A Fast Unified Model for Parsing and Sentence Understanding
Published 1 January 2016Open access
Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, Christopher Potts
Citations236
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The Stack-augmentedParser-Interpreter NeuralNetwork (SPINN) combines parsing and interpretation within a single tree-sequence hybrid model by integrating tree-structured sentence interpretation into the linear sequential structure of a shiftreduceparser.
Abstract
Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, Christopher Potts. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2016.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Dropout: a simple way to prevent neural networks from overfitting
34,279 Citations2014Nitish Srivastava, Geoffrey E. Hinton +3 more
It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
arXiv (Cornell University)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
24,381 Citations2015Sergey Ioffe, Christian Szegedy
Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
18,777 Citations2015Kaiming He, Xiangyu Zhang +2 more
This work proposes a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit and derives a robust initialization method that particularly considers the rectifier nonlinearities.
arXiv (Cornell University)Neural Machine Translation by Jointly Learning to Align and Translate
14,565 Citations2014Dzmitry Bahdanau
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
A Convolutional Neural Network for Modelling Sentences
3,561 Citations2014Nal Kalchbrenner, Edward Grefenstette +1 more
A convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) is described that is adopted for the semantic modelling of sentences and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations.
A large annotated corpus for learning natural language inference
3,452 Citations2015Samuel R. Bowman, Gabor Angeli +2 more
The Stanford Natural Language Inference corpus is introduced, a new, freely available collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning, which allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time.
arXiv (Cornell University)Character-level Convolutional Networks for Text Classification
3,266 Citations2015Xiang Zhang, Junbo Zhao +1 more
This article constructed several large-scale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results in text classification.
A Fast and Accurate Dependency Parser using Neural Networks
1,879 Citations2014Danqi Chen, Christopher D. Manning
This work proposes a novel way of learning a neural network classifier for use in a greedy, transition-based dependency parser that can work very fast, while achieving an about 2% improvement in unlabeled and labeled attachment scores on both English and Chinese datasets.
arXiv (Cornell University)Static Analysis of Shape in TensorFlow Programs
1,669 Citations2020The Theano Development Team, Rami Al‐Rfou +3 more
The performance of Theano is compared against Torch7 and TensorFlow on several machine learning models and recently-introduced functionalities and improvements are discussed.
Lecture notes in computer scienceThe PASCAL Recognising Textual Entailment Challenge
1,626 Citations2006Ido Dagan, Oren Glickman +1 more
The Theory of Parsing, Translation, and Compiling
1,411 Citations1972Alfred V. Aho, Jeffrey D. Ullman
It is the hope that the algorithms and concepts presented in this book will survive the next generation of computers and programming languages, and that at least some of them will be applicable to fields other than compiler writing.
arXiv (Cornell University)Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
1,268 Citations2015Samy Bengio, Oriol Vinyals +2 more
Parsing Natural Scenes and Natural Language with Recursive Neural Networks
1,209 Citations2011Richard Socher, Cliff Chiung-Yu Lin +2 more
A max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences is introduced.
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
1,198 Citations2011Richard Socher, Jeffrey Pennington +3 more
A novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions that outperform other state-of-the-art approaches on commonly used datasets, without using any pre-defined sentiment lexica or polarity shifting rules.
Long Short-Term Memory-Networks for Machine Reading
1,071 Citations2016Jianpeng Cheng, Li Dong +1 more
A machine reading simulator which processes text incrementally from left to right and performs shallow reasoning with memory and attention and extends the Long Short-Term Memory architecture with a memory network in place of a single memory cell, offering a way to weakly induce relations among tokens.
arXiv (Cornell University)Theano: new features and speed improvements
1,035 Citations2012Frédéric Bastien, Pascal Lamblin +7 more
New features and efficiency improvements to Theano are presented, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.
Proceedings of International Conference on Neural Networks (ICNN'96)Learning task-dependent distributed representations by backpropagation through structure
592 Citations2002Christoph Goller, Andreas Küchler
Transition-Based Dependency Parsing with Stack Long Short-Term Memory
560 Citations2015Chris Dyer, Miguel Ballesteros +3 more
This work was sponsored in part by the U. S. Army Research Laboratory and the NSF CAREER grant IIS-1054319 and the European Commission.
An efficient algorithm for projective dependency parsing
497 Citations2003Joakim Nivre
This paper presents a deterministic parsing algorithm for projective dependency grammar that has been experimentally evaluated in parsing unrestricted Swedish text, achieving an accuracy above 85% with a very simple grammar.
Learning Natural Language Inference with LSTM
408 Citations2016Shuohang Wang, Jing Jiang
A special long short-term memory (LSTM) architecture for NLI that remembers important mismatches that are critical for predicting the contradiction or the neutral relationship label and achieves an accuracy of 86.1%, outperforming the state of the art.
arXiv (Cornell University)Reasoning about Entailment with Neural Attention
407 Citations2015Rockt\"aschel, Tim, Edward Grefenstette +3 more
Natural Language Inference by Tree-Based Convolution and Heuristic Matching
369 Citations2016Lili Mou, Rui Men +5 more
This model, a tree-based convolutional neural network (TBCNN) captures sentence-level semantics; then heuristic matching layers like concatenation, element-wise product/difference combine the information in individual sentences.
Journal of Functional ProgrammingThe Zipper
310 Citations1997Gérard Huet
The Zipper is Huet's nifty name for a nifty data structure which fulfills this need of representing a tree together with a subtree that is the focus of attention, where that focus may move left, right, up or down the tree.
neural information processing systemsDeep Recursive Neural Networks for Compositionality in Language
259 Citations2014Ozan İrsoy, Claire Cardie
The results show that deep RNNs outperform associated shallow counterparts that employ the same number of parameters and outperforms previous baselines on the sentiment analysis task, including a multiplicative RNN variant as well as the recently introduced paragraph vectors.
arXiv (Cornell University)Long Short-Term Memory-Networks for Machine Reading
190 Citations2016Jianpeng Cheng, Li Dong +1 more
arXiv (Cornell University)Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
178 Citations2015Armand Joulin, Tomáš Mikolov
When Are Tree Structures Necessary for Deep Learning of Representations?
162 Citations2015Jiwei Li, Thang Luong +2 more
This paper benchmarks recursive neural models against sequential recurrent neural models, enforcing applesto-apples comparison as much as possible, and introduces a method for allowing recurrent models to achieve similar performance: breaking long sentences into clause-like units at punctuation and processing them separately before combining.
arXiv (Cornell University)Learning to Transduce with Unbounded Memory
138 Citations2015Edward Grefenstette, Karl Moritz Hermann +2 more
This paper proposes new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues and shows that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in the transduction experiments.
Compositionality as an Empirical Problem
138 Citations2007David R. Dowty
arXiv (Cornell University)When Are Tree Structures Necessary for Deep Learning of Representations?
135 Citations2015Jiwei Li, Minh-Thang Luong +2 more
Discriminative training of a neural network statistical parser
110 Citations2004James Henderson
It is shown how a parser can be trained with a discriminative learning method while still parameterizing the problem according to a generative probability model, resulting in state-of-the-art levels of performance.
Text, speech and language technologyA Latent Variable Model for Generative Dependency Parsing
81 Citations2010Ivan Titov, James Henderson
This work proposes a generative dependency parsing model which uses binary latent variables to induce conditioning features and uses a recently proposed class of Bayesian Networks for structured prediction, Incremental Sigmoid Belief Networks to define this model.
Top-down Tree Long Short-Term Memory Networks
76 Citations2016Xingxing Zhang, Liang Lu +1 more
This paper develops Tree Long Short-Term Memory (TreeLSTM), a neural network model based on LSTM, which is designed to predict a tree rather than a linear sequence, and reports results on dependency parsing reranking achieving competitive performance.
Sentence disambiguation by a shift-reduce parsing technique
69 Citations1983Stuart M. Shieber
A parsing algorithm is developed--a variant of the LALR(I) shift-reduce algorithm--that models the preference behavior of native speakers for a range of syntactic preference phenomena reported in the psycholinguistic literature, including the recent data on lexical preferences.
Machine LearningA Neural Syntactic Language Model
67 Citations2005Ahmad Emami, Frederick Jelinek
The neural syntactic based model achieves the best published results in perplexity and WER for the given data sets and comparisons with the standard and neural net based N-gram models with arbitrarily long contexts show that the syntactic information is in fact very helpful in estimating the word string probability.
arXiv (Cornell University)Tree-structured composition in neural networks without tree-structured architectures
41 Citations2015Samuel R. Bowman, Christopher D. Manning +1 more
It is found that an LSTM-based sequence model can indeed learn to exploit the underlying tree structure, and its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.
arXiv (Cornell University)Recognizing Entailment and Contradiction by Tree-based Convolution
19 Citations2015Lili Mou, Rui Men +5 more
Experimental results on a large dataset verify the rationale of using TBCNN as the sentencelevel model; leveraging additional heuristics like element-wise product/difference further improves the accuracy.
Generative Incremental Dependency Parsing with Neural Networks
17 Citations2015Jan Buys, Phil Blunsom
A neural network model for scalable generative transition-based dependency parsing that surpasses the accuracy and speed of previous generative dependency parsers and shows a strong improvement over n-gram language models, opening the way to the efficient integration of syntax into neural models for language generation.
Transactions of the Association for Computational LinguisticsEasy-First Dependency Parsing with Hierarchical Tree LSTMs
11 Citations2016Eliyahu Kiperwasser, Yoav Goldberg
A compositional vector representation of parse trees that relies on a recursive combination of recurrent-neural network encoders is suggested, achieving very strong accuracies for English and Chinese, without relying on external word embeddings.