Deep Semantic Role Labeling: What Works and What’s Next
Published 1 January 2017Open access
Luheng He, Kenton Lee, Mike Lewis, Luke Zettlemoyer
Citations451
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new deep learning model for semantic role labeling (SRL) that significantly improves the state of the art, along with detailed analyses to reveal its strengths and limitations are introduced.
Abstract
We introduce a new deep learning model for semantic role labeling (SRL) that significantly improves the state of the art, along with detailed analyses to reveal its strengths and limitations. We use a deep highway BiLSTM architecture with constrained decoding, while observing a number of recent best practices for initialization and regularization. Our 8-layer ensemble model achieves 83.2 F1 on the CoNLL 2005 test set and 83.4 F1 on CoNLL 2012, roughly a 10% relative error reduction over the previous state of the art. Extensive empirical analysis of these gains show that (1) deep models excel at recovering long-distance dependencies but can still make surprisingly obvious errors, and (2) that there is still room for syntactic parsers to improve these results.
Keywords
Computer Science
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
arXiv (Cornell University)ADADELTA: An Adaptive Learning Rate Method
5,531 Citations2012Matthew D. Zeiler
A novel per-dimension learning rate method for gradient descent called ADADELTA that dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent is presented.
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
Neural ComputationTraining Products of Experts by Minimizing Contrastive Divergence
5,017 Citations2002Geoffrey E. Hinton
A product of experts (PoE) is an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary because it is hard even to approximate the derivatives of the renormalization term in the combination rule.
Natural Language Processing (almost) from Scratch
3,987 Citations2011Ronan Collobert, Jason Weston +4 more
A maximum-entropy-inspired parser
1,496 Citations2000Eugene Charniak
A new parser for parsing down to Penn tree-bank style parse trees that achieves 90.1% average precision/recall for sentences of length 40 and less and 89.5% when trained and tested on the previously established sections of the Wall Street Journal treebank is presented.
arXiv (Cornell University)Training Very Deep Networks
1,100 Citations2015Rupesh K. Srivastava, Klaus Greff +1 more
arXiv (Cornell University)Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
1,014 Citations2013Andrew Saxe, James L. McClelland +1 more
Oxford University Research Archive (ORA) (University of Oxford)A theoretically grounded application of dropout in recurrent neural networks
914 Citations2016Yarin Gal, Zoubin Ghahramani
arXiv (Cornell University)A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
782 Citations2015Yarin Gal, Zoubin Ghahramani
This work applies a new variational inference based dropout technique in LSTM and GRU models, which outperforms existing techniques, and to the best of the knowledge improves on the single model state-of-the-art in language modelling with the Penn Treebank.
Introduction to the CoNLL-2005 shared task
649 Citations2005Xavier Carreras, Lluı́s Màrquez
The specification and goals of the task are introduced, the data sets and evaluation methods are described, and a general overview of the 19 systems that have contributed to the task is presented, providing a comparative description and results.
arXiv (Cornell University)Training Very Deep Networks
546 Citations2015Rupesh K. Srivastava, Klaus Greff +1 more
A new architecture designed to overcome the challenges of training very deep networks, inspired by Long Short-Term Memory recurrent networks, which allows unimpeded information flow across many layers on information highways.
Computational LinguisticsThe Importance of Syntactic Parsing and Inference in Semantic Role Labeling
413 Citations2008Vasin Punyakanok, Dan Roth +1 more
It is shown that full syntactic parsing information is, by far, most relevant in identifying the argument, especially in the very first stagethe pruning stage, and an effective and simple approach of combining different semantic role labeling systems through joint inference is proposed, which significantly improves its performance.
Introduction to the CoNLL-2004 Shared Task: Semantic Role Labeling
396 Citations2004Xavier Carreras, Lluı́s Màrquez
The specification and goal of the task are introduced, the data sets and evaluation methods are described, and a general overview of the systems that have contributed to the task is presented, providing comparative description.
End-to-end learning of semantic role labeling using recurrent neural networks
366 Citations2015Jie Zhou, Wei Xu
This work proposes to use deep bi-directional recurrent network as an end-to-end system for SRL, which takes only original text information as input feature, without using any syntactic knowledge.
Institutional Research Information System (Università degli Studi di Trento)Towards Robust Linguistic Analysis using OntoNotes
362 Citations2013Sameer Pradhan, Alessandro Moschitti +6 more
An analysis of the performance of publicly available, state-of-the-art tools on all layers and languages in the OntoNotes v5.0 corpus should set the benchmark for future development of various NLP components in syntax and semantics, and possibly encourage research towards an integrated system that makes use of the various layers jointly to improve overall performance.
Highway long short-term memory RNNS for distant speech recognition
299 Citations2016Yu Zhang, Guoguo Chen +4 more
This paper extends the deep long short-term memory (DL-STM) recurrent neural networks by introducing gated direct connections between memory cells in adjacent layers, and introduces the latency-controlled bidirectional LSTMs (BLSTMs) which can exploit the whole history while keeping the latency under control.
Neural Semantic Role Labeling with Dependency Path Embeddings
192 Citations2016Michael Roth, Mirella Lapata
A novel model for semantic role labeling that makes use of neural sequence modeling techniques and treats complex syntactic structures and related phenomena, such as nested subordinations and nominal predicates, as subsequences of lexicalized dependency paths and learns suitable embedding representations.
Computational LinguisticsA Global Joint Model for Semantic Role Labeling
147 Citations2008Kristina Toutanova, Aria Haghighi +1 more
A model for semantic role labeling that effectively captures the linguistic intuition that a semantic argument frame is a joint structure, with strong dependencies among the arguments, and how to incorporate these strong dependencies in a statistical joint model with a rich set of features over multiple argument phrases is presented.
Semantic Role Labeling with Neural Network Factors
131 Citations2015Nicholas FitzGerald, Oscar Täckström +2 more
A new method for semantic role labeling in which arguments and semantic roles are jointly embedded in a shared vector space for a given predicate, which is based on a neural network designed for the SRL task.
Generalized inference with multiple semantic role labeling systems
130 Citations2005P. Koomen, Vasin Punyakanok +2 more
An approach to semantic role labeling (SRL) that takes the output of multiple argument classifiers and combines them into a coherent predicate-argument output by solving an optimization problem is presented.
A* CCG Parsing with a Supertag-factored Model
115 Citations2014Mike Lewis, Mark Steedman
A new CCG parsing model is introduced which is factored on lexical category assignments, which allows a highly effective heuristic for A parsing, which makes parsing extremely fast.
Transactions of the Association for Computational LinguisticsEfficient Inference and Structured Learning for Semantic Role Labeling
83 Citations2015Oscar Täckström, Kuzman Ganchev +1 more
A dynamic programming algorithm for efficient constrained inference in semantic role labeling that tractably captures a majority of the structural constraints examined by prior work in this area, and allows training a globally-normalized log-linear model with respect to constrained conditional likelihood.
Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output
83 Citations2012Jonathan K. Kummerfeld, David Hall +2 more
This work classifies errors within a set of linguistically meaningful types using tree transformations that repair groups of errors together, and uses this analysis to answer a range of questions about parser behaviour, including what linguistic constructions are difficult for state-of-the-art parsers, what types of errors are being resolved by rerankers, and what types are introduced when parsing out- of-domain text.
Semantic role chunking combining complementary syntactic views
73 Citations2005Sameer Pradhan, Kadri Hacıoğlu +3 more
This paper describes a semantic role labeling system that uses features derived from different syntactic views, and combines them within a phrase-based chunking paradigm, and gives a significant improvement in performance over roles produced by using any one of the syntactic view individually.
Computational LinguisticsMultilingual Joint Parsing of Syntactic and Semantic Dependencies with a Latent Variable Model
70 Citations2013James Henderson, Paola Merlo +2 more
A joint generative history-based model to predict the most likely derivation of a dependency parser for both syntactic and semantic dependencies, in multiple languages, and proposes a weak synchronization at the level of meaningful subsequences of the two derivations.
Chinese Semantic Role Labeling with Bidirectional Recurrent Neural Networks
39 Citations2015Zhen Wang, Tingsong Jiang +2 more
Bidirectional recurrent neural network with long-short-term memory (LSTM) with RNN to capture bidirectional and long-range dependencies in a sentence with minimal feature engineering is introduced.
A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling
29 Citations2017Diego Marcheggiani, Anton Frolov +1 more
A simple and accurate neural model for dependency-based semantic role labeling that predicts predicate-argument dependencies relying on states of a bidirectional LSTM encoder that substantially outperforms all previous local models and approaches the best reported results on the English CoNLL-2009 dataset.