Towards Robust Linguistic Analysis using OntoNotes
Institutional Research Information System (Università degli Studi di Trento)Published 1 August 2013Open access
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina
Citations362
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An analysis of the performance of publicly available, state-of-the-art tools on all layers and languages in the OntoNotes v5.0 corpus should set the benchmark for future development of various NLP components in syntax and semantics, and possibly encourage research towards an integrated system that makes use of the various layers jointly to improve overall performance.
Abstract
Large-scale linguistically annotated corpora have played a crucial role in advancing the state of the art of key natural language technologies such as syntactic, semantic and discourse analyzers, and they serve as training data as well as evaluation benchmarks. Up till now, however, most of the evaluation has been done on monolithic corpora such as the Penn Treebank, the Proposition Bank. As a result, it is still unclear how the state-of-the-art analyzers perform in general on data from a variety of genres or domains. The completion of the OntoNotes corpus, a large-scale, multi-genre, multilingual corpus manually annotated with syntactic, semantic and discourse information, makes it possible to perform such an evaluation. This paper presents an analysis of the performance of publicly available, state-of-the-art tools on all layers and languages in the OntoNotes v5.0 corpus. This should set the benchmark for future development of various NLP components in syntax and semantics, and possibly encourage research towards an integrated system that makes use of the various layers jointly to improve overall performance. 1
Keywords
Computer Science
Building a Large Annotated Corpus of English: The Penn Treebank
7,528 Citations1993Mitchell P. Marcus
Incorporating non-local information into information extraction systems by Gibbs sampling
3,035 Citations2005Jenny Rose Finkel, Trond Grenager +1 more
By using simulated annealing in place of Viterbi decoding in sequence models such as HMMs, CMMs, and CRFs, it is possible to incorporate non-local structure while preserving tractable inference.
Computational LinguisticsThe Proposition Bank: An Annotated Corpus of Semantic Roles
2,302 Citations2005Martha Palmer, Daniel Gildea +1 more
An automatic system for semantic role tagging trained on the corpus is described and the effect on its performance of various types of information is discussed, including a comparison of full syntactic parsing with a flat representation and the contribution of the empty trace categories of the treebank.
The Penn Discourse TreeBank 2.0.
1,130 Citations2008Rashmi Prasad, Nikhil Dinesh +5 more
The second version of the Penn Discourse Treebank, PDTB-2.0, is presented, describing its lexically-grounded annotations of discourse relations and their two abstract object arguments over the 1 million word Wall Street Journal corpus.
Coarse-to-fine <i>n</i>-best parsing and MaxEnt discriminative reranking
888 Citations2005Eugene Charniak, Mark Johnson
This paper describes a simple yet novel method for constructing sets of 50- best parses based on a coarse-to-fine generative parser that generates 50-best lists that are of substantially higher quality than previously obtainable.
Improving machine learning approaches to coreference resolution
709 Citations2001Vincent Ng, Claire Cardie
A noun phrase coreference system that extends the work of Soon et al. (2001) and produces the best results to date on the M UC-6 and MUC-7 coreference resolution data sets --- F-measures of 70.4 and 63.4, respectively.
Natural Language EngineeringThe Penn Chinese TreeBank: Phrase structure annotation of a large corpus
694 Citations2005Nianwen Xue, Fei Xia +2 more
Several Chinese linguistic issues and their implications for treebanking efforts are discussed and how to address these issues when developing annotation guidelines are addressed, and engineering strategies to improve speed while ensuring annotation quality are described.
Institutional Research Information System (Università degli Studi di Trento)CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes
669 Citations2012Sameer Pradhan, Alessandro Moschitti +3 more
The OntoNotes annotation (coreference and other layers) is described and the parameters of the shared task including the format, pre-processing information, evaluation criteria, and presents and discusses the results achieved by the participating systems.
Introduction to the CoNLL-2005 shared task
649 Citations2005Xavier Carreras, Lluı́s Màrquez
The specification and goals of the task are introduced, the data sets and evaluation methods are described, and a general overview of the 19 systems that have contributed to the task is presented, providing a comparative description and results.
Effective self-training for parsing
619 Citations2006David McClosky, Eugene Charniak +1 more
This work presents a simple, but surprisingly effective, method of self-training a two-phase parser-reranker system using readily available unlabeled data and shows that this type of bootstrapping is possible for parsing when the bootstrapped parses are processed by a discriminative reranker.
Improved Inference for Unlexicalized Parsing
610 Citations2007Slav Petrov, Dan Klein
A novel coarse-to-fine method in which a grammar’s own hierarchical projections are used for incremental pruning, including a method for efficiently computing projections of a grammar without a treebank is presented.
National University of SingaporeIt Makes Sense: A Wide-Coverage Word Sense Disambiguation System for Free Text
286 Citations2010Zhi Zhong, Hwee Tou Ng
The flexible framework of IMS allows users to integrate different preprocessing tools, additional features, and different classifiers, and it achieves state-of-the-art results on several SensEval and SemEval tasks.
Machine LearningSupport Vector Learning for Semantic Argument Classification
278 Citations2005Sameer Pradhan, Kadri Hacıoğlu +4 more
A machine learning algorithm for semantic role parsing is proposed, extending the work of Gildea and Jurafsky (2002), Surdeanu et al. (2003) and others, based on Support Vector Machines which shows large improvement in performance over earlier classifiers.
CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes
273 Citations2011Sameer Pradhan, Lance Ramshaw +4 more
The CoNLL-2011 shared task involved predicting coreference using OntoNotes data, a new resource that provides multiple integrated annotation layers (parses, semantic roles, word senses, named entities and coreference) that could support joint models.
Corpus Variation and Parser Performance
273 Citations2001Daniel Gildea
This work examines how other types of text might affect parser performance, and how portable parsing models are across corpora by comparing results for the Brown and WSJ corpora, and considers which parts of the parser’s probability model are particularly tuned to the corpus on which it was trained.
Joint parsing and named entity recognition
192 Citations2009Jenny Rose Finkel, Christopher D. Manning
This work proposes a joint model of parsing and named entity recognition, based on a discriminative feature-based constituency parser that produces a consistent output, where the named entity spans do not conflict with the phrasal spans of the parse tree.
Edit detection and parsing for transcribed speech
158 Citations2001Eugene Charniak, Mark Johnson
A simple architecture for parsing transcribed speech in which an edited-word detector first removes such words from the sentence string, and then a standard statistical parser trained on transcribedspeech parses the remaining words is presented.
Natural Language EngineeringMaking fine-grained and coarse-grained sense distinctions, both manually and automatically
155 Citations2006Martha Palmer, Hoa Trang Dang +1 more
A persistent problem arising from polysemy is discussed: namely the difficulty of finding consistent criteria for making fine-grained sense distinctions, either manually or automatically, and well-defined sense groups can be of value in improving word sense disambiguation by both humans and machines.
SemEval-2010 Task 1: Coreference Resolution in Multiple Languages
150 Citations2010Marta Vilar Recasens, Lluı́s Màrquez +6 more
This paper presents the SemEval-2010 task on Coreference Resolution in Multiple Languages, which was to evaluate and compare automatic coreference resolution systems for six different languages in four evaluation settings and using four different metrics.
Computational LinguisticsLabeling Chinese Predicates with Semantic Roles
135 Citations2008Nianwen Xue
This article reports work on Chinese semantic role labeling, taking advantage of two recently completed corpora, the Chinese PropBank, a semantically annotated corpus of Chinese verbs, and the Chinese Nombank, a companion corpus that annotates the predicate-argument structure of nominalized predicates.
Bootstrapping path-based pronoun resolution
131 Citations2006Shane Bergsma, Dekang Lin
This work learns the likelihood of coreference between a pronoun and a candidate noun based on the path in the parse tree between the two entities, and robustly addresses traditional syntactic coreference constraints.
The Life and Death of Discourse Entities: Identifying Singleton Mentions
122 Citations2013Marta Vilar Recasens, Marie-Catherine de Marneffe +1 more
A logistic regression model is built for predicting the singleton/coreferent distinction, drawing on linguistic insights about how discourse entity lifespans are affected by syntactic and semantic features.
Computational LinguisticsTowards Robust Semantic Role Labeling
119 Citations2008Sameer Pradhan, Wayne Ward +1 more
The operation of assert, a state-of-the art SRL system, is described and the robustness of the system when trained on one genre of data and used to label a different genre is analyzed.
International Conference on Semantic Computing (ICSC 2007)Elements of a National SemanticWeb Infrastructure--Case Study Finland on the Semantic Web
117 Citations2007Eero Hyvönen, Robin Lindroos +14 more
Natural Language EngineeringAdding semantic roles to the Chinese Treebank
104 Citations2008Nianwen Xue, Martha Palmer
This discussion focuses on the syntactic variations in the realization of the arguments as well as the approach to annotating dislocated and discontinuous arguments, and the creation of a lexical database of frame files and its role in guiding predicate-argument annotation.
Acquiring Compact Lexicalized Grammars from a Cleaner Treebank
101 Citations2002Julia Hockenmaier, Mark Steedman
An algorithm which translates the Penn Treebank into a corpus of Combinatory Categorial Grammar (CCG) derivations is presented and it is demonstrated how the combination of preprocessing and type-changing rules minimizes the lexical coverage problem.
Developing an Arabic treebank
94 Citations2004Mohamed Maamouri, Ann Bies
This paper addresses the following questions from the experience of developing a large-scale corpus of Arabic text annotated for morphological information, part-of-speech, English gloss, and syntactic structure.
An empirical study of the domain dependence of supervised word sense disambiguation systems
59 Citations2000Gerard Escudero, Lluı́s Màrquez +1 more
It is demonstrated empirically that the Lazy-Boosting algorithm outperforms state-of-the-art supervised WSD algorithms in both previous situations.
Evaluation Metrics For End-to-End Coreference Resolution Systems
56 Citations2010Jie Cai, Michael Strube
Two variants of the B3 and CEAF coreferenceresolution evaluation algorithms which can be applied to coreference resolution systems dealing with system mentions are proposed, i.e. automatically determined mentions.
Language-Independent Parsing with Empty Elements
49 Citations2011Shu Cai, David Chiang +1 more
A simple, language-independent method for integrating recovery of empty elements into syntactic parsing that outperforms the best published method the authors are aware of on English and a recently published method on Chinese.
Data-driven Multilingual Coreference Resolution using Resolver Stacking
41 Citations2012Anders Björkelund, Richárd Farkas
A novel decoding algorithm for coreference resolution which is combined with a standard pair-wise coreference resolver in a stacking approach is presented and obtained an official overall score of 58.25 which is the second highest in the Shared Task.
Chasing the ghost: recovering empty categories in the Chinese Treebank
39 Citations2010Yaqin Yang, Nianwen Xue
A unified framework in recovering empty categories in the Chinese Tree-bank is described and the results show that given skeletal gold standard parses, the empty categories can be detected with very high accuracy.
Linguistic Annotation WorkshopThe Revised Arabic PropBank
35 Citations2010Wajdi Zaghouani, Mona Diab +3 more
Near-infrared electroluminescence from ambipolar, electrolyte-gated arrays of highly aligned single-walled carbon nanotubes (SWNT) is demonstrated and introducing thin layers of HfO(2) and TiO( 2) provides a means to modify exciton screening without fundamentally changing the current-voltage characteristics or electrolumscence yield of these devices.
Issues in synchronizing the English Treebank and PropBank
27 Citations2006Olga Babko-Malaya, Ann Bies +6 more
The types of mismatches between the syntactic bracketing and the semantic role labeling that can be found are discussed, and the plans for reconciling them are discussed.
Computers and the HumanitiesLTAG-spinal and the Treebank
23 Citations2007Libin Shen, Lucas Champollion +1 more
LTAG-spinal is introduced, a novel variant of traditional Lexicalized Tree Adjoining Grammar with desirable linguistic, computational and statistical properties and provides a very desirable resource for statistical LTAG parsing, incremental parsing, dependency parsing, and semantic parsing.
Word sense disambiguation using OntoNotes
22 Citations2008Zhi Zhong, Hwee Tou Ng +1 more
This work conducts the first large-scale WSD evaluation involving hundreds of word types and tens of thousands of sense-tagged examples, while adopting a coarse-grained sense inventory and proposes a domain adaptation technique using feature augmentation with active learning.
Semantic Role Labeling Systems for Arabic using Kernel Methods
22 Citations2008Mona Diab, Alessandro Moschitti +1 more
This paper presents an SRL system for Modern Standard Arabic that exploits many aspects of the rich morphological features of the language and improves the current state-of-the-art in Arabic SRL.
Evaluation of semantic role labeling and dependency parsing of automatic speech recognition output
8 Citations2010Benoît Favre, Bernd Bohnet +1 more
This work extends the standard evaluation metrics for joint dependency parsing and SRL of text in order to be able to handle speech recognition output with word errors and sentence segmentation errors.
ProQuest LLC eBooksNull element restoration
7 Citations2010Mitch Marcus, Ryan Gabbard
This thesis presents a system for inserting null elements into parse trees in English, and examines the application of information derived from the Google Web 1T corpus as a way of reducing certain data sparsity issues related to wh-movement.
Singular or Plural? Exploiting Parallel Corpora for Chinese Number Prediction.
5 Citations2011Elizabeth Baran, Nianwen Xue