Enhanced LSTM for Natural Language Inference
Published 1 January 2017Open access
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, Diana Inkpen
Citations1,184
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper presents a new state-of-the-art result, achieving the accuracy of 88.6% on the Stanford Natural Language Inference Dataset, and demonstrates that carefully designing sequential inference models based on chain LSTMs can outperform all previous models.
Abstract
Reasoning and inference are central to human and artificial intelligence.\nModeling inference in human language is very challenging. With the availability\nof large annotated data (Bowman et al., 2015), it has recently become feasible\nto train neural network based inference models, which have shown to be very\neffective. In this paper, we present a new state-of-the-art result, achieving\nthe accuracy of 88.6% on the Stanford Natural Language Inference Dataset.\nUnlike the previous top models that use very complicated network architectures,\nwe first demonstrate that carefully designing sequential inference models based\non chain LSTMs can outperform all previous models. Based on this, we further\nshow that by explicitly considering recursive architectures in both local\ninference modeling and inference composition, we achieve additional\nimprovement. Particularly, incorporating syntactic parsing information\ncontributes to our best result---it further improves the performance even when\nadded to the already very strong model.\n
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
UvA-DARE (University of Amsterdam)Adam: A Method for Stochastic Optimization
84,783 Citations2014Diederik P. Kingma, Jimmy Ba
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)
50,318 Citations2021Mandi, Jayanta, Canoy, Rocsildes +2 more
A simple numeric simulation of DNA-co-polymerized hydrogel shape change and a genetic algorithm that generates and selects large batches of material designs that compete with one another to evolve and converge on optimal objective-matching designs are constructed.
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
arXiv (Cornell University)Neural Machine Translation by Jointly Learning to Align and Translate
14,565 Citations2014Dzmitry Bahdanau
arXiv (Cornell University)Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
7,525 Citations2015Kelvin Xu, Jimmy Ba +6 more
An attention based model that automatically learns to describe the content of images is introduced that can be trained in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound.
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
6,774 Citations2013Richard Socher, Alex Perelygin +5 more
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
6,616 Citations2014Kyunghyun Cho, Bart van Merriënboer +2 more
It is shown that the neural machine translation performs relatively well on short sentences without unknown words, but its performance degrades rapidly as the length of the sentence and the number of unknown words increase.
A large annotated corpus for learning natural language inference
3,452 Citations2015Samuel R. Bowman, Gabor Angeli +2 more
The Stanford Natural Language Inference corpus is introduced, a new, freely available collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning, which allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time.
Accurate unlexicalized parsing
3,055 Citations2003Dan Klein, Christopher D. Manning
It is demonstrated that an unlexicalized PCFG can parse much more accurately than previously shown, by making use of simple, linguistically motivated state splits, which break down false independence assumptions latent in a vanilla treebank grammar.
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
2,308 Citations2016William Chan, Navdeep Jaitly +2 more
Listen, Attend and Spell (LAS), a neural speech recognizer that transcribes speech utterances directly to characters without pronunciation models, HMMs or other components of traditional speech recognizers is presented.
arXiv (Cornell University)Attention-Based Models for Speech Recognition
1,780 Citations2015Jan Chorowski, Dzmitry Bahdanau +3 more
The attention-mechanism is extended with features needed for speech recognition and a novel and generic method of adding location-awareness to the attention mechanism is proposed to alleviate the issue of high phoneme error rate.
arXiv (Cornell University)Show, Attend and Tell: Neural Image Caption Generation with Visual\n Attention
1,754 Citations2015Kelvin Xu, Jimmy Ba +6 more
Lecture notes in computer scienceThe PASCAL Recognising Textual Entailment Challenge
1,626 Citations2006Ido Dagan, Oren Glickman +1 more
A Decomposable Attention Model for Natural Language Inference
1,372 Citations2016Ankur P. Parikh, Oscar Täckström +2 more
This work proposes a simple neural architecture for natural language inference that uses attention to decompose the problem into subproblems that can be solved separately, thus making it trivially parallelizable.
Parsing Natural Scenes and Natural Language with Recursive Neural Networks
1,209 Citations2011Richard Socher, Cliff Chiung-Yu Lin +2 more
A max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences is introduced.
Long Short-Term Memory-Networks for Machine Reading
1,071 Citations2016Jianpeng Cheng, Li Dong +1 more
A machine reading simulator which processes text incrementally from left to right and performs shallow reasoning with memory and attention and extends the Long Short-Term Memory architecture with a memory network in place of a single memory cell, offering a way to weakly induce relations among tokens.
arXiv (Cornell University)A large annotated corpus for learning natural language inference
465 Citations2015Samuel R. Bowman, Gabor Angeli +2 more
arXiv (Cornell University)Reasoning about Entailment with Neural Attention
427 Citations2015Tim Rocktäschel, Edward Grefenstette +3 more
This paper proposes a neural model that reads two sentences to determine entailment using long short-term memory units and extends this model with a word-by-word neural attention mechanism that encourages reasoning over entailments of pairs of words and phrases, and presents a qualitative analysis of attention weights produced by this model.
Learning Natural Language Inference with LSTM
408 Citations2016Shuohang Wang, Jing Jiang
A special long short-term memory (LSTM) architecture for NLI that remembers important mismatches that are critical for predicting the contradiction or the neutral relationship label and achieves an accuracy of 86.1%, outperforming the state of the art.
arXiv (Cornell University)Reasoning about Entailment with Neural Attention
407 Citations2015Rockt\"aschel, Tim, Edward Grefenstette +3 more
Natural Language Inference by Tree-Based Convolution and Heuristic Matching
369 Citations2016Lili Mou, Rui Men +5 more
This model, a tree-based convolutional neural network (TBCNN) captures sentence-level semantics; then heuristic matching layers like concatenation, element-wise product/difference combine the information in individual sentences.
Attention-Based Models for Speech Recognition
299 Citations2015Jan Chorowski, Dzmitry Bahdanau +3 more
National Research Council Canada (Government of Canada)Long Short-Term Memory Over Recursive Structures
272 Citations2015Xiaodan Zhu, Parinaz Sobihani +1 more
This paper proposes to extend chain-structured long short-term memory to tree structures, in which a memory cell can reflect the history memories of multiple child cells or multiple descendant cells in a recursive process, and calls the model S-LSTM, which provides a principled way of considering long-distance interaction over hierarchies.
A Fast Unified Model for Parsing and Sentence Understanding
236 Citations2016Samuel R. Bowman, Jon Gauthier +4 more
The Stack-augmentedParser-Interpreter NeuralNetwork (SPINN) combines parsing and interpretation within a single tree-sequence hybrid model by integrating tree-structured sentence interpretation into the linear sequential structure of a shiftreduceparser.
arXiv (Cornell University)Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
234 Citations2016Yang Liu, Chengjie Sun +2 more
A sentence encoding-based model for recognizing text entailment that utilized the sentence's first-stage representation to attend words appeared in itself, which is called "Inner-Attention" in this paper.
Natural language inference
223 Citations2009Christopher D. Manning, Bill MacCartney
This dissertation explores a range of approaches to NLI, beginning with methods which are robust but approximate, and proceeding to progressively more precise approaches, and greatly extends past work in natural logic to incorporate both semantic exclusion and implicativity.
arXiv (Cornell University)Long Short-Term Memory-Networks for Machine Reading
190 Citations2016Jianpeng Cheng, Li Dong +1 more
Modeling semantic containment and exclusion in natural language inference
170 Citations2008Bill MacCartney, Christopher D. Manning
This work proposes an approach to natural language inference based on a model of natural logic, which identifies valid inferences by their lexical and syntactic features, without full semantic interpretation, to incorporate both semantic exclusion and implicativity.
arXiv (Cornell University)Order-Embeddings of Images and Language
87 Citations2015Ivan Vendrov, Ryan Kiros +2 more
An Empirical Study on the Effect of Negation Words on Sentiment
79 Citations2014Xiaodan Zhu, Hongyu Guo +2 more
A sentiment treebank is used to show that existing heuristics used to estimate the sentiment of negated expressions are poor estimators of sentiment and a recently proposed composition model that relies on both the negator and the argument is evaluated.
International Joint Conference on Artificial IntelligenceDistraction-based neural networks for modeling documents
79 Citations2016Chen Qian, Xiaodan Zhu +3 more
This paper proposes neural models to train computers not just to pay attention to specific regions and content of input documents with attention models, but also distract them to traverse between different content of a document so as to better grasp the overall meaning for summarization.
Compositional Distributional Semantics with Long Short Term Memory
69 Citations2015Phong Ba Le, Willem Zuidema
An extension of the recursive neural network that makes use of a variant of the long short-term memory architecture that provides a solution to the vanishing gradient problem and allows the network to capture long range dependencies.
International Conference on Computational LinguisticsReading and Thinking: Re-read LSTM Unit for Textual Entailment Recognition
62 Citations2016Lei Sha, Baobao Chang +2 more
A new L STM unit called re-read LSTM (rLSTM) is designed to implement this “thinking” process, which means to read the premise again and again while reading the hypothesis to get a better understanding of the premise.
Revisiting Word Embedding for Contrasting Meaning
46 Citations2015Zhigang Chen, Wei Lin +5 more
This paper presents the embedding models that achieve an F-score of 92% on the widely-used, publicly available dataset, the GRE “most contrasting word” questions, and examines several basic concerns in modeling contrasting meaning to provide detailed analysis.
arXiv (Cornell University)Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
39 Citations2017Junbei Zhang, Xiaodan Zhu +4 more
This paper first introduces syntactic information to help encode questions and then view and model different types of questions and the information shared among them as an adaptation task and proposed adaptation models for them.
Cineca Institutional Research Information System (Tor Vergata University)Syntactic/Semantic Structures for Textual Entailment Recognition
29 Citations2010Yashar Mehdad, Alessandro Moschitti +1 more
An approach based on off-the-shelf parsers and semantic resources for the Recognizing Textual Entailment (RTE) challenge that can be generally applied to any domain and consistently achieves high accuracy, without requiring any adaptation or tuning.
Neural Tree Indexers for Text Understanding
16 Citations2017Tsendsuren Munkhdalai, Hong Yu
A robust syntactic parsing-independent tree structured model, Neural Tree Indexers (NTI) is introduced that provides a middle ground between the sequential RNNs and the syntactic treebased recursive models, outperforming state-of-the-art recurrent and recursive neural networks.
Cineca Institutional Research Information System (Tor Vergata University)Towards Syntax-aware Compositional Distributional Semantic Models
12 Citations2014Lorenzo Ferrone, Fabio Massimo Zanzotto
It is shown that under a suitable regime these two approaches can be regarded as the same and, thus, structural information and distributional semantics can successfully cooperate in CSDMs for NLP tasks.
arXiv (Cornell University)A Neural Architecture Mimicking Humans End-to-End for Natural Language\n Inference
9 Citations2016Biswajit Paria, K. M. Annervaz +3 more
arXiv (Cornell University)A Neural Architecture Mimicking Humans End-to-End for Natural Language Inference
8 Citations2016Biswajit Paria, K. M. Annervaz +3 more
This work uses the recent advances in representation learning to propose a neural architecture for the problem of natural language inference that achieves better accuracy numbers than all published models in literature.