Annotating Temporal Dependency Graphs via Crowdsourcing
Published 1 January 2020Open access
Jiarui Yao, Haoling Qiu, Bonan Min, Nianwen Xue
Citations12
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is argued that temporal dependency graphs, built on previous research on narrative times and temporal anaphora, provide a representation scheme that achieves a good trade-off between completeness and practicality in temporal annotation.
Abstract
We present the construction of a corpus of 500 Wikinews articles annotated with temporal dependency graphs (TDGs) that can be used to train systems to understand temporal relations in text. We argue that temporal dependency graphs, built on previous research on narrative times and temporal anaphora, provide a representation scheme that achieves a good trade-off between completeness and practicality in temporal annotation. We also provide a crowdsourcing strategy to annotate TDGs, and demonstrate the feasibility of this approach with an evaluation of the quality of the annotation, and the utility of the resulting data set by training a machine learning model on this data set. The data set is publicly available.
Keywords
Computer Science
The Stanford CoreNLP Natural Language Processing Toolkit
7,238 Citations2014Christopher D. Manning, Mihai Surdeanu +4 more
The design and use of the Stanford CoreNLP toolkit is described, an extensible pipeline that provides core natural language analysis, and it is suggested that this follows from a simple, approachable design, straightforward interfaces, the inclusion of robust and good quality analysis components, and not requiring use of a large amount of associated baggage.
Cheap and fast---but is it good?
1,906 Citations2008Rion Snow, Brendan O’Connor +2 more
This work explores the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web, and proposes a technique for bias correction that significantly improves annotation quality on two tasks.
A Fast and Accurate Dependency Parser using Neural Networks
1,879 Citations2014Danqi Chen, Christopher D. Manning
This work proposes a novel way of learning a neural network classifier for use in a greedy, transition-based dependency parser that can work very fast, while achieving an about 2% improvement in unlabeled and labeled attachment scores on both English and Chinese datasets.
International Journal of Molecular SciencesTimeML: Robust Specification of Event and Temporal Expressions in Text
717 Citations2003James Pustejovsky, José M. Castaño +6 more
This review focuses on the regulatory role of CEACAM1 in liver and biliary tract architecture in health and disease, and on its metabolic role and function as an immune checkpoint regulator of hepatic inflammation.
SemEval-2013 Task 1: TempEval-3: Evaluating Time Expressions, Events, and Temporal Relations
360 Citations2013Naushad UzZaman, Héctor Llorens +4 more
The participants’ approaches, results, and the observations from the results, which may guide future research in this area are described.
ScholarlyCommons (University of Pennsylvania)TENSE AS DISCOURSE ANAPHOR
236 Citations1988Bonnie Lynn Webber
Language Resources and EvaluationMultilingual and cross-domain temporal tagging
214 Citations2012Jannik Strötgen, Michael Gertz
The authors' publicly available temporal tagger HeidelTime is presented, which is easily extensible to further languages due to its strict separation of source code and language resources like patterns and rules.
An Annotation Framework for Dense Event Ordering
162 Citations2014Taylor Cassidy, Bill McDowell +2 more
This paper proposes a new annotation process with a mechanism to force annotators to label connected graphs that generates 10 times more relations per document than the TimeBank, and its TimeBank-Dense corpus is larger than all current corpora.
A Multi-Axis Annotation Scheme for Event Temporal Relations
127 Citations2018Ning Qiang, Hao Wu +1 more
A new multi-axis modeling to better capture the temporal structure of events is proposed and it is identified that event end-points are a major source of confusion in annotation, so it is proposed to annotate TempRels based on start-points only.
arXiv (Cornell University)TempEval-3: Evaluating Events, Time Expressions, and Temporal Relations
85 Citations2012Naushad UzZaman, Héctor Llorens +4 more
The TempEval-3 task is described, incorporating a three-part task structure covering event, temporal expression and temporal relation extraction; a larger dataset; and single overall task quality scores.
arXiv (Cornell University)A Multi-Axis Annotation Scheme for Event Temporal Relations
71 Citations2018Ning Qiang, Hao Wu +1 more
Increasing Informativeness in Temporal Annotation
65 Citations2011James Pustejovsky, Amber Stubbs
A document level information structure the authors call a narrative container (NC) is introduced, designed to increase informativeness and accuracy of temporal relation identification, in the creation of a new temporal annotation over TimeBank.
International Conference on Semantic Computing (ICSC 2007)Learning by Reading by Learning to Read
55 Citations2007Sergei Nirenburg, Tim Oates +1 more
The architecture of the GIBRALTAR system, which is based on the OntoSem semantic analyzer, is described, which learns by reading by learning to read and extracts knowledge about the topics of the texts and knowledge about language that enables improved text processing.
Lirias (KU Leuven)Extracting Narrative Timelines as Temporal Dependency Structures
48 Citations2012Oleksandr Kolomiyets, Steven Bethard +1 more
This work annotates a corpus of children's stories with temporal dependency trees, and shows that a deterministic non-projective dependency parser outperforms a graph-based maximum spanning tree parser, achieving labeled attachment accuracy and labeled tree edit distance.
Temporal Anchoring of Events for the TimeBank Corpus
41 Citations2016Nils Reimers, Nazanin Dehghani +1 more
This paper proposes a new annotation scheme to anchor events in time that is much lower as it scales linear with the number of events, and gives a more precise anchoring when the events have happened as the complete document can be taken into account.
International Conference on Semantic Computing (ICSC 2007)Timelines from Text: Identification of Syntactic Temporal Relations
24 Citations2007Steven Bethard, James Martin +1 more
Neural Ranking Models for Temporal Dependency Structure Parsing
15 Citations2018Yuchen Zhang, Nianwen Xue
The first neural temporal dependency parser is designed and built, utilizes a neural ranking model with minimal feature engineering, and parses time expressions and events in a text into a temporal dependency tree structure, and beats two strong baselines on both data domains.
Improved Temporal Relation Classification using Dependency Parses and Selective Crowdsourced Annotations
8 Citations2012Jun-Ping Ng, Min‐Yen Kan
Acquiring Structured Temporal Representation via Crowdsourcing: A Feasibility Study
6 Citations2019Yuchen Zhang, Nianwen Xue
This paper introduces a method to crowdsource temporal dependency tree annotations, and shows that this representation is intuitive and can be collected with high accuracy and agreement through crowdsourcing.
Improved Temporal Relation Classification using Dependency Parses and Selective Crowdsourced Annotations
4 Citations2012Ping Ng Min
β-cat gene expression is positively correlated with recurrence, distant metastasis, and tumor-node-metastasis stage and is found that the patients had a better survival if they had a lower β- cat gene expression.