Feature Weight Tuning for Recursive Neural Networks
arXiv (Cornell University)Published 11 December 2014Open access
Jiwei Li
Citations13
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper addresses how a recursive neural network model can automatically leave out useless information and emphasize important evidence, in other words, to perform "weight tuning" for higher-level representation acquisition.
Abstract
This paper addresses how a recursive neural network model can automatically leave out useless information and emphasize important evidence, in other words, to perform "weight tuning" for higher-level representation acquisition. We propose two models, Weighted Neural Network (WNN) and Binary-Expectation Neural Network (BENN), which automatically control how much one specific unit contributes to the higher-level representation. The proposed model can be viewed as incorporating a more powerful compositional function for embedding acquisition in recursive neural networks. Experimental results demonstrate the significant improvement over standard neural models.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
24,447 Citations2014Kyunghyun Cho, Bart van Merriënboer +5 more
Qualitatively, the proposed RNN Encoder‐Decoder model learns a semantically and syntactically meaningful representation of linguistic phrases.
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
11,710 Citations2013Tomáš Mikolov, Kai Chen +2 more
IEEE Transactions on Signal ProcessingBidirectional recurrent neural networks
10,022 Citations1997Mike Schuster, Kuldip K. Paliwal
It is shown how the proposed bidirectional structure can be easily modified to allow efficient estimation of the conditional posterior probability of complete symbol sequences without making any explicit assumption about the shape of the distribution.
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
8,612 Citations2011John C. Duchi, Elad Hazan +1 more
Thumbs up?
6,987 Citations2002Bo Pang, Lillian Lee +1 more
This work considers the problem of classifying documents not by topic, but by overall sentiment, e.g., determining whether a review is positive or negative, and concludes by examining factors that make the sentiment classification problem more challenging.
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
6,774 Citations2013Richard Socher, Alex Perelygin +5 more
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
Neural ComputationLearning to Forget: Continual Prediction with LSTM
5,419 Citations2000Felix A. Gers, Jürgen Schmidhuber +1 more
This work identifies a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset, and proposes a novel, adaptive forget gate that enables an LSTm cell to learn to reset itself at appropriate times, thus releasing internal resources.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
Neural ComputationA Learning Algorithm for Continually Running Fully Recurrent Neural Networks
4,451 Citations1989Ronald J. Williams, David Zipser
The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal supervised learning tasks.
A Convolutional Neural Network for Modelling Sentences
3,561 Citations2014Nal Kalchbrenner, Edward Grefenstette +1 more
A convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) is described that is adopted for the semantic modelling of sentences and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations.
Learning Word Vectors for Sentiment Analysis
3,299 Citations2011Andrew L. Maas, Raymond E. Daly +4 more
This work presents a model that uses a mix of unsupervised and supervised techniques to learn word vectors capturing semantic term--document information as well as rich sentiment content, and finds it out-performs several previously introduced methods for sentiment classification.
Semantic Compositionality through Recursive Matrix-Vector Spaces
1,297 Citations2012Richard Socher, Brody Huval +2 more
A recursive neural network model that learns compositional vector representations for phrases and sentences of arbitrary syntactic type and length and can learn the meaning of operators in propositional logic and natural language is introduced.
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
1,198 Citations2011Richard Socher, Jeffrey Pennington +3 more
A novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions that outperform other state-of-the-art approaches on commonly used datasets, without using any pre-defined sentiment lexica or polarity shifting rules.
Generating Text with Recurrent Neural Networks
1,166 Citations2011Ilya Sutskever, James Martens +1 more
The power of RNNs trained with the new Hessian-Free optimizer by applying them to character-level language modeling tasks is demonstrated, and a new RNN variant that uses multiplicative connections which allow the current input character to determine the transition matrix from one hidden state vector to the next is introduced.
Parsing with Compositional Vector Grammars
832 Citations2013Richard Socher, John Bauer +2 more
A Compositional Vector Grammar (CVG), which combines PCFGs with a syntactically untied recursive neural network that learns syntactico-semantic, compositional vector representations and improves performance on the types of ambiguities that require semantic information such as PP attachments.
Computational LinguisticsModeling Local Coherence: An Entity-Based Approach
687 Citations2008Regina Barzilay, Mirella Lapata
This article re-conceptualize coherence assessment as a learning task and shows that the proposed entity-grid representation of discourse is well-suited for ranking-based generation and text classification tasks.
Proceedings of International Conference on Neural Networks (ICNN'96)Learning task-dependent distributed representations by backpropagation through structure
592 Citations2002Christoph Goller, Andreas Küchler
Three new graphical models for statistical language modelling
582 Citations2007Andriy Mnih, Geoffrey E. Hinton
It is shown how real-valued distributed representations for words can be learned at the same time as learning a large set of stochastic binary hidden features that are used to predict the distributed representation of the next word from previous distributed representations.
Neural Information Processing SystemsSubsequence Kernels for Relation Extraction
507 Citations2005Raymond J. Mooney, Răzvan Bunescu
A new kernel method for extracting semantic relations between entities in natural language text, based on a generalization of subsequence kernels, is presented, which uses three types of subsequent patterns that are typically employed innatural language to assert relationships between two entities.
arXiv (Cornell University)Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization
299 Citations2004Regina Barzilay, Lillian Lee
An effective knowledge-lean method for learning content models from unannotated documents is presented, utilizing a novel adaptation of algorithms for Hidden Markov Models and applied to two complementary tasks: information ordering and extractive summarization.
Representation Learning for Text-level Discourse Parsing
242 Citations2014Yangfeng Ji, Jacob Eisenstein
A representation learning approach, in which surface features are transformed into a latent space that facilitates RST discourse parsing, which obtains substantial improvements over the previous state-of-the-art in predicting relations and nuclearity on the RST Treebank.
Translation Modeling with Bidirectional Recurrent Neural Networks
182 Citations2014Martin Sundermeyer, Tamer Alkhouli +2 more
This work presents phrase-based translation models that are more consistent with phrasebased decoding and introduces bidirectional recurrent neural models to the problem of machine translation, allowing the full source sentence to be used in the models.
A Model of Coherence Based on Distributed Sentence Representation
158 Citations2014Jiwei Li, Eduard Hovy
A neural network model for the coherence task based on distributed sentence representation is introduced, which outperforms existing baselines and generates the state-of-art performance in standard coherence evaluation tasks.
Recursive Deep Models for Discourse Parsing
149 Citations2014Jiwei Li, Rumeng Li +1 more
This paper proposes a recursive model for discourse parsing that jointly models distributed representations for clauses, sentences, and entire discourses that obtains comparable performance regarding standard discoursing parsing evaluations when compared against current state-of-art systems.
arXiv (Cornell University)Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network
91 Citations2014Misha Denil, Alban Demiraj +3 more
A model is introduced that is able to represent the meaning of documents by embedding them in a low dimensional vector space, while preserving distinctions of word and sentence order crucial for capturing nuanced semantics.
A Coherence Model Based on Syntactic Patterns
76 Citations2012Annie Louis, Ani Nenkova
A model of coherence which captures the intentional discourse structure in text based on the hypothesis that syntax provides a proxy for the communicative goal of a sentence and therefore the sequence of sentences in a coherent discourse should exhibit detectable structural patterns.
arXiv (Cornell University)Bidirectional Recursive Neural Networks for Token-Level Labeling with Structure
17 Citations2013Ozan İrsoy, Claire Cardie
This work proposes a novel architecture that aims to capture the structural information around an input, and use it to label instances, and applies it to the task of opinion expression extraction.