Deep Learning for Efficient Discriminative Parsing
Published 14 June 2011Open access
Ronan Collobert
Citations157
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new fast purely discriminative algorithm for natural language parsing, based on a “deep” recurrent convolutional graph transformer network (GTN) and assuming a decomposition of a parse tree into a stack of "levels”, the network predicts a level of the tree taking into account predictions of previous levels.
Abstract
We propose a new fast purely discriminative algorithm for natural language parsing, based on a “deep ” recurrent convolutional graph transformer network (GTN). Assuming a decomposition of a parse tree into a stack of “levels”, the network predicts a level of the tree taking into account predictions of previous levels. Using only few basic text features, we show similar performance (in F1 score) to existing pure discriminative parsers and existing “benchmark ” parsers (like Collins parser, probabilistic context-free grammars based), with a huge speed advantage. 1
Keywords
Computer Science
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
12,978 Citations2001John Lafferty, Andrew McCallum +1 more
This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
Neural ComputationBackpropagation Applied to Handwritten Zip Code Recognition
11,878 Citations1989Yann LeCun, Bernhard E. Boser +5 more
This paper demonstrates how constraints from the task domain can be integrated into a backpropagation network through the architecture of the network, successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service.
Building a Large Annotated Corpus of English: The Penn Treebank
7,528 Citations1993Mitchell P. Marcus
IEEE Transactions on Information TheoryError bounds for convolutional codes and an asymptotically optimum decoding algorithm
6,705 Citations1967Andrew J. Viterbi
The upper bound is obtained for a specific probabilistic nonsequential decoding algorithm which is shown to be asymptotically optimum for rates above R_{0} and whose performance bears certain similarities to that of sequential decoding algorithms.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
Computational LinguisticsHead-Driven Statistical Models for Natural Language Parsing
1,860 Citations2003Michael Collins
Three statistical models for natural language parsing are described, leading to approaches in which a parse tree is represented as the sequence of decisions corresponding to a head-centered, top-down derivation of the tree.
Applied Physics Letters10.1162/153244303322533223
1,771 Citations2000
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences.
A maximum-entropy-inspired parser
1,496 Citations2000Eugene Charniak
A new parser for parsing down to Penn tree-bank style parse trees that achieves 90.1% average precision/recall for sentences of length 40 and less and 89.5% when trained and tested on the previously established sections of the Wall Street Journal treebank is presented.
Shallow parsing with conditional random fields
1,251 Citations2003Fei Sha, Fernando Pereira
This work shows how to train a conditional random field to achieve performance as good as any reported base noun-phrase chunking method on the CoNLL task, and better than any reported single model.
Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons
1,161 Citations2003Andrew McCallum, Wei Li
This work has shown that conditionally-trained models, such as conditional maximum entropy models, handle inter-dependent features of greedy sequence modeling in NLP well.
A Neural Probabilistic Language Model
1,156 Citations2000Yoshua Bengio, Réjean Ducharme +1 more
Coarse-to-fine <i>n</i>-best parsing and MaxEnt discriminative reranking
888 Citations2005Eugene Charniak, Mark Johnson
This paper describes a simple yet novel method for constructing sets of 50- best parses based on a coarse-to-fine generative parser that generates 50-best lists that are of substantially higher quality than previously obtainable.
A new statistical parser based on bigram lexical dependencies
619 Citations1996Michael Collins
A new statistical parser which is based on probabilities of dependencies between head-words in the parse tree, which trains on 40,000 sentences in under 15 minutes and can be improved to over 200 sentences a minute with negligible loss in accuracy.
Stochastic Gradient Learning in Neural Networks
565 Citations1991Léon Bottou
This chapter discusses connectionist learning algorithms, which consists of minimizing a cost of the form C(w) = E(J(z,w)dP(z) where dP is an unknown probability distribution that characterizes the problem to learn, and J, the loss function, defines the learning system itself.
Statistical decision-tree models for parsing
533 Citations1995David M. Magerman
SPATTER is described, a statistical parser based on decision-tree learning techniques which constructs a complete parse for every sentence and achieves accuracy rates far better than any published result.
Natural Language EngineeringPartial parsing via finite-state cascades
480 Citations1996Steven Abney
Deterministic parsers specified by finite state cascades may be more accurate than exhaustive search stochastic context free parsers and extended at modest cost to construct parse trees with finite feature structures.
Machine LearningLearning to Parse Natural Language with Maximum Entropy Models
245 Citations1999Adwait Ratnaparkhi
A machine learning system for parsing natural language that learns from manually parsed example sentences, and parses unseen data at state-of-the-art accuracies, and it is demonstrated that the parser can train from other domains without modification to the modeling framework or the linguistic hints it uses to learn.
Empirical Methods in Natural Language ProcessingMax-Margin Parsing
198 Citations2004Ben Taskar, Dan Klein +3 more
A novel discriminative approach to parsing inspired by the large-margin criterion underlying support vector machines is presented, which allows one to efficiently learn a model which discriminates among the entire space of parse trees, as opposed to reranking the top few candidates.
Efficient, Feature-based, Conditional Random Field Parsing
180 Citations2008Jenny Rose Finkel, Alex Kleeman +1 more
This work presents the first general, featurerich discriminative parser, based on a conditional random field model, which has been successfully scaled to the full WSJ parsing data, and achieves state-of-the-art results.
Computer Speech & LanguageLearning sets of filters using back-propagation
115 Citations1987David C. Plaut, Geoffrey E. Hinton
Further research is described on back-propagation for layered networks of deterministic, neuron-like units and an example in which a network learns a set of filters that enable it to discriminate formant-like patterns in the presence of noise.
TAG, dynamic programming, and the perceptron for efficient, feature-rich parsing
114 Citations2008Xavier Carreras, Michael J. Collins +1 more
A parsing approach that makes use of the perceptron algorithm, in conjunction with dynamic programming methods, to recover full constituent-based parse trees, using a Tree Adjoining Grammar (TAG) based parsing formalism.
Discriminative training of a neural network statistical parser
110 Citations2004James Henderson
It is shown how a parser can be trained with a discriminative learning method while still parameterizing the problem according to a generative probability model, resulting in state-of-the-art levels of performance.
Semantic role labelling with tree conditional random fields
94 Citations2005Trevor Cohn, Philip Blunsom
This paper shows how modelling the semantic role labelling task as a tree labelling problem allows for the use of efficient CRF inference algorithms, while also increasing generalisation performance when compared to the equivalent maximum entropy classifier.
Global training of document processing systems using graph transformer networks
93 Citations2002Léon Bottou, Yoshua Bengio +1 more
A new machine learning paradigm called Graph Transformer Networks is proposed that extends the applicability of gradient-based learning algorithms to systems composed of modules that take graphs as inputs and produce graphs as output.
Sparse multi-scale grammars for discriminative latent variable parsing
38 Citations2008Slav Petrov, Dan Klein
This work presents a discriminative, latent variable approach to syntactic parsing in which rules exist at multiple scales of refinement, and produces the best published parsing accuracies with the smallest reported grammars.
Advances in discriminative parsing
26 Citations2006Joseph Turian, I. Dan Melamed
This work has no generative component, yet surpasses a generative baseline on constituent parsing, and does so with minimal linguistic cleverness, and performs feature selection incrementally over an exponential feature space during training.