Towards Robust Semantic Role Labeling
Computational LinguisticsPublished 1 June 2008Open access
Sameer Pradhan, Wayne Ward, James Martin
Citations119
SJR quartileQ1
SJR score1.15
SNIP4.16
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The operation of assert, a state-of-the art SRL system, is described and the robustness of the system when trained on one genre of data and used to label a different genre is analyzed.
Abstract
Most semantic role labeling (SRL) research has been focused on training and evaluating on the same corpus. This strategy, although appropriate for initiating research, can lead to overtraining to the particular corpus. This article describes the operation of assert, a state-of-the art SRL system, and analyzes the robustness of the system when trained on one genre of data and used to label a different genre. As a starting point, results are first presented for training and testing the system on the PropBank corpus, which is annotated Wall Street Journal (WSJ) data. Experiments are then presented to evaluate the portability of the system to another source of data. These experiments are based on comparisons of performance using PropBanked WSJ data and PropBanked Brown Corpus data. The results indicate that whereas syntactic parses and argument identification transfer relatively well to a new corpus, argument classification does not. An analysis of the reasons for this is presented and these generally point to the nature of the more lexical/semantic features dominating the classification task where more general structural features are dominant in the argument identification task.
Keywords
Computer Science
The Nature of Statistical Learning Theory
39,279 Citations1995Vladimir Vapnik
TechnometricsStatistical Learning Theory
26,913 Citations1999Yuhai Wu, Vladimir Vapnik
Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
Data Mining and Knowledge DiscoveryA Tutorial on Support Vector Machines for Pattern Recognition
16,433 Citations1998Christopher J. C. Burges
There are several arguments which support the observed high accuracy of SVMs, which are reviewed and numerous examples and proofs of most of the key theorems are given.
Lecture notes in computer scienceText categorization with Support Vector Machines: Learning with many relevant features
7,925 Citations1998Thorsten Joachims
SVMs achieve substantial improvements over the currently best performing methods and behave robustly over a variety of di-erent learning tasks, eliminating the need for manual parameter tuning.
Building a Large Annotated Corpus of English: The Penn Treebank
7,528 Citations1993Mitchell P. Marcus
IEEE ExpertData mining and knowledge discovery: making sense out of data
4,643 Citations1996U.M. Feyyad
Find loads of the data mining and knowledge discovery making sense out of data book catalogues in this site as the choice of you visiting this page.
Computational LinguisticsThe Proposition Bank: An Annotated Corpus of Semantic Roles
2,302 Citations2005Martha Palmer, Daniel Gildea +1 more
An automatic system for semantic role tagging trained on the corpus is described and the effect on its performance of various types of information is discussed, including a comparison of full syntactic parsing with a flat representation and the contribution of the empty trace categories of the treebank.
Computational LinguisticsAutomatic Labeling of Semantic Roles
1,647 Citations2002Daniel Gildea, Daniel Jurafsky
Automatic retrieval and clustering of similar words
1,555 Citations1998Dekang Lin
A word similarity measure based on the distributional pattern of words allows the automatically constructed thesaurus to be significantly closer to WordNet than Roget Thesaurus is.
A maximum-entropy-inspired parser
1,496 Citations2000Eugene Charniak
A new parser for parsing down to Penn tree-bank style parse trees that achieves 90.1% average precision/recall for sentences of length 40 and less and 89.5% when trained and tested on the previously established sections of the Wall Street Journal treebank is presented.
Text, speech and language technologyText Chunking Using Transformation-Based Learning
1,257 Citations1999Lance Ramshaw, Mitchell P. Marcus
This work has shown that the transformation-based learning approach can be applied at a higher level of textual interpretation for locating chunks in the tagged text, including non-recursive “baseNP” chunks.
Coarse-to-fine <i>n</i>-best parsing and MaxEnt discriminative reranking
888 Citations2005Eugene Charniak, Mark Johnson
This paper describes a simple yet novel method for constructing sets of 50- best parses based on a coarse-to-fine generative parser that generates 50-best lists that are of substantially higher quality than previously obtainable.
Wiley Encyclopedia of Electrical and Electronics EngineeringComputational Linguistics
834 Citations1999Fred Popowich
Machine LearningAn Algorithm that Learns What's in a Name
784 Citations1999Daniel M. Bikel, Richard Schwartz +1 more
IdentiFinderTM, a hidden Markov model that learns to recognize and classify names, dates, times, and numerical quantities, is evaluated and is competitive with approaches based on handcrafted rules on mixed case text and superior on text where case information is not available.
The Penn Treebank
751 Citations1994Mitchell P. Marcus, Grace Kim +6 more
The implementation of crucial aspects of this new syntactic annotation scheme incorporates a more consistent treatment of a wide range of grammatical phenomena, provides a set of coindexed null elements in what can be thought of as "underlying" position for phenomena such as wh-movement, passive, and the subjects of infinitival constructions.
Introduction to the CoNLL-2005 shared task
649 Citations2005Xavier Carreras, Lluı́s Màrquez
The specification and goals of the task are introduced, the data sets and evaluation methods are described, and a general overview of the 19 systems that have contributed to the task is presented, providing a comparative description and results.
Effective self-training for parsing
619 Citations2006David McClosky, Eugene Charniak +1 more
This work presents a simple, but surprisingly effective, method of self-training a two-phase parser-reranker system using readily available unlabeled data and shows that this type of bootstrapping is possible for parsing when the bootstrapped parses are processed by a discriminative reranker.
Applied Physics Letters10.1162/153244302760200687
534 Citations2000
An approximation technique is introduced that is shown to deliver good approximations efficiently for large datasets and is shown to deliver good approximations efficiently for large datasets.
Chunking with support vector machines
533 Citations2001Taku Kudo, Yūji Matsumoto
Introduction to the CoNLL-2004 Shared Task: Semantic Role Labeling
396 Citations2004Xavier Carreras, Lluı́s Màrquez
The specification and goal of the task are introduced, the data sets and evaluation methods are described, and a general overview of the systems that have contributed to the task is presented, providing comparative description.
Using predicate-argument structures for information extraction
370 Citations2003Mihai Surdeanu, Sanda M. Harabagiu +2 more
The experimental results prove the claim that accurate predicate-argument structures enable high quality IE results, and introduce a new way of automatically identifying predicate argument structures, which is central to the IE paradigm.
Shallow Semantic Parsing using Support Vector Machines.
368 Citations2004Sameer Pradhan, Wayne Ward +3 more
A machine learning al- algorithm for shallow semantic parsing based on Support Vector Machines which shows performance improvements through a number of new features and their ability to generalize to a new test set drawn from the AQUAINT corpus is measured.
Calibrating Features for Semantic Role Labeling
320 Citations2004Nianwen Xue, Martha Palmer
A critical look at the features used in the semantic role tagging literature is taken and it is shown that the information in the input, generally a syntactic parse tree, has yet to be fully exploited.
Use of support vector learning for chunk identification
285 Citations2000Taku Kudoh, Yūji Matsumoto
This paper investigates how SVMs with a very large number of features perform with the classification task of chunk labelling, CoNLL-2000 shared task, chunk identification.
Machine LearningSupport Vector Learning for Semantic Argument Classification
278 Citations2005Sameer Pradhan, Kadri Hacıoğlu +4 more
A machine learning algorithm for semantic role parsing is proposed, extending the work of Gildea and Jurafsky (2002), Surdeanu et al. (2003) and others, based on Support Vector Machines which shows large improvement in performance over earlier classifiers.
Corpus Variation and Parser Performance
273 Citations2001Daniel Gildea
This work examines how other types of text might affect parser performance, and how portable parsing models are across corpora by comparing results for the Brown and WSJ corpora, and considers which parts of the parser’s probability model are particularly tuned to the corpus on which it was trained.
Reranking and self-training for parser adaptation
241 Citations2006David McClosky, Eugene Charniak +1 more
The reranking parser described in Charniak and Johnson (2005) improves performance of the parser on Brown to 85.2% and use of the self-training techniques described in (McClosky et al., 2006) raise this to 87.8% (an error reduction of 28%) again without any use of labeled Brown data.
Question answering based on semantic structures
214 Citations2004Srini Narayanan, Sanda M. Harabagiu
The ability of the implemented system to perform several forms of probabilistic and temporal inferences to extract answers to complex questions is reported on, indicating enhanced accuracy over current state-of-the-art Q/A systems.
Generative models for statistical parsing with Combinatory Categorial Grammar
209 Citations2001Julia Hockenmaier, Mark Steedman
This paper compares a number of generative probability models for a wide-coverage Combinatory Categorial Grammar (CCG) parser and finds a significant improvement from modeling word-word dependencies.
Joint learning improves semantic role labeling
182 Citations2005Kristina Toutanova, Aria Haghighi +1 more
This work shows how to build a joint model of argument frames, incorporating novel features that model these interactions into discriminative log-linear models, achieving error reduction of 22% on all arguments and 32% on core arguments over a state-of-the art independent classifier for gold-standard parse trees on PropBank.
Journal of the American Statistical AssociationVariable Selection in Data Mining
152 Citations2004Dean P. Foster, Robert A. Stine
This work shows that three modifications turn stepwise regression into an effective methodology for predicting bankruptcy, and predicts bankruptcy as well as, if not better than, recently developed data-mining tools.
DSpace@MIT (Massachusetts Institute of Technology)Statistical Models for Co-occurrence Data
145 Citations1998Thomas Hofmann, Jan Puzicha
A statistical framework for analyzing co-occurrence data in a general setting where elementary observations are joint occurrences of pairs of abstract objects from two finite sets is developed and a novel family of mixture models is proposed.
Semantic role labeling using different syntactic views
136 Citations2005Sameer Pradhan, Wayne Ward +3 more
A state-of-the-art baseline semantic role labeling system based on Support Vector Machine classifiers is presented and improvements on this system are shown by adding new features including features extracted from dependency parses, performing feature selection and calibration and combining parses obtained from semantic parsers trained using different syntactic views.
Generalized inference with multiple semantic role labeling systems
130 Citations2005P. Koomen, Vasin Punyakanok +2 more
An approach to semantic role labeling (SRL) that takes the output of multiple argument classifiers and combines them into a coherent predicate-argument output by solving an optimization problem is presented.
Learning and inference over constrained output
105 Citations2005Vasin Punyakanok, Dan Roth +2 more
Experiments and theoretical justification lead to the conclusion that using inference based learning is superior when the local classifiers are difficult to learn but may require many examples before any discernible difference can be observed.
Acquiring Compact Lexicalized Grammars from a Cleaner Treebank
101 Citations2002Julia Hockenmaier, Mark Steedman
An algorithm which translates the Penn Treebank into a corpus of Combinatory Categorial Grammar (CCG) derivations is presented and it is demonstrated how the combination of preprocessing and type-changing rules minimizes the lexical coverage problem.
Identifying semantic roles using Combinatory Categorial Grammar
99 Citations2003Daniel Gildea, Julia Hockenmaier
A system for automatically identifying PropBank-style semantic roles based on the output of a statistical parser for Combinatory Categorial Grammar, and outperforms it on core argument roles.
Semantic Role Labeling by Tagging Syntactic Chunks
96 Citations2004Kadri Hacıoğlu, Sameer Pradhan +3 more
A semantic role la-beler (or chunker) that groups syntactic chunks into the arguments of a predicate by casting the semantic labeling as the classification of syntactic chunks into one of several classes such as the beginning of an ar-gument, inside an argument and outside an argument.
Semantic role parsing: adding semantic structure to unstructured text
86 Citations2004Sameer Pradhan, Kadri Hacıoğlu +3 more
Bilingual FrameNet Dictionaries for Machine Translation
81 Citations2002Hans C. Boas
The goals of GFN are to create lexical entries for German nouns, verbs, and adjectives that correspond to existing FrameNet entries, and to link the parallel lexicon fragments by means of common semantic frames and numerical indexing mechanisms.
Computer Speech & LanguageMAP adaptation of stochastic grammars
74 Citations2005Michiel Bacchiani, Michael Riley +2 more
It is shown that the commonly used approaches of count merging and model interpolation are special cases of a more general maximum a posteriori (MAP) framework, which additionally allows for alternate adaptation approaches.
How verb subcategorization frequencies are affected by corpus choice
74 Citations1998Douglas Roland, Daniel Jurafsky
A joint model for semantic role labeling
56 Citations2005Aria Haghighi, Kristina Toutanova +1 more
A semantic role labeling system submitted to the closed track of the CoNLL-2005 shared task implements a joint model that captures dependencies among arguments of a predicate using log-linear models in a discriminative re-ranking framework.
International Joint Conference on Artificial IntelligenceShallow semantics for relation extraction
56 Citations2005Sanda M. Harabagiu, Cosmin A. Bejan +1 more
A new method for extracting meaningful relations from unstructured natural language sources based on information made available by shallow semantic parsers that surpassed the results of kernel-based models employing only semantic class information.
The integration of syntactic parsing and semantic role labeling
31 Citations2005Szu-ting Yi, Martha Palmer
This paper describes a system for the CoNLL-2005 Shared Task on Semantic Role Labeling that trained two parsers with the training corpus in which the semantic argument information is attached to the constituent labels, and used the resulting parse trees as the input of the pipelined SRL system.
A robust combination strategy for semantic role labeling
22 Citations2005Lluı́s Màrquez, Mihai Surdeanu +2 more
A simple and robust strategy for system combination is presented, which allows to partially recover from input parsing errors and to significantly boost results of individual systems.
Syntactic kernels for natural language learning
21 Citations2006Alessandro Moschitti
This paper uses tree kernels to exploit deep syntactic parsing information for natural language applications and provides algorithms for their computation in linear average time.
Automatic construction of an English-Chinese bilingual FrameNet
19 Citations2004Benfeng Chen, Pascale Fung
This work coerces the English FrameNet into Chinese using a bilingual lexicon, frame context in FrameNet and taxonomy structure in HowNet, and achieves a promising 82% average F-measure for the most ambiguous lexical entries.
Accurate parsing of the proposition bank
18 Citations2006Gabriele Musillo, Paola Merlo
This work integrates PropBank semantic role labels to an existing statistical parsing model producing richer output and shows conclusive results on joint learning and inference of syntactic and semantic representations.
A lightweight semantic chunking model based on tagging
18 Citations2004Kadri Hacıoğlu
A framework for the development of a fast, accurate, and highly portable semantic chunker is introduced based on a non-overlapping, shallow tree-structured language and results are presented that show that the phrase-by-phrase approach performs better than its word- by-word counterpart.
ScopusSemantic argument classification exploiting argument interdependence
12 Citations2005Zheng Ping Jiang, Li Jia +1 more
This paper proposes the use of the neighboring semantic arguments of a predicate as additional features in determining the class of the current semantic argument, and shows significant improvement in the accuracy of semantic argument classification after exploiting argument interdependence.