SemEval-2013 Task 1: TempEval-3: Evaluating Time Expressions, Events, and Temporal Relations
Published 1 June 2013
Naushad UzZaman, Héctor Llorens, Leon Derczynski, James F. Allen, Marc Verhagen, James Pustejovsky
Citations360
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The participants’ approaches, results, and the observations from the results, which may guide future research in this area are described.
Abstract
Within the SemEval-2013 evaluation exercise, the TempEval-3 shared task aims to advance research on temporal information processing. It follows on from TempEval-1 and-2, with: a three-part structure covering temporal expression, event, and temporal relation extraction; a larger dataset; and new single measures to rank systems – in each task and in general. In this paper, we describe the participants’ approaches, results, and the observations from the results, which may guide future research in this area. 1
Keywords
Computer Science
Communications of the ACMMaintaining knowledge about temporal intervals
7,550 Citations1983James F. Allen
An interval-based temporal logic is introduced, together with a computationally effective reasoning algorithm based on constraint propagation, which is notable in offering a delicate balance between space and time.
Journal of the American Medical Informatics AssociationAgreement, the F-Measure, and Reliability in Information Retrieval
938 Citations2005George Hripcsak
It can be shown that the average F-measure among pairs of experts is numerically identical to the average positive specific agreement among experts and that kappa approaches these measures as the number of negative cases grows large.
FreeLing 3.0: Towards Wider Multilinguality
491 Citations2012Lluís Padró, Evgeny Stanilovsky
The general architecture of the library is described, the major changes and improvements included in FreeLing version 3.0 are presented, and some relevant industrial projects in which it has been used are summarized.
PLoS Computational BiologyGetting More Out of Biomedical Documents with GATE's Full Lifecycle Open Source Text Analytics
363 Citations2013Hamish Cunningham, Valentin Tablan +2 more
It is concluded that the deployment of text mining for document abstraction or rich search and navigation is best thought of as a process, and that with the right computational tools and data collection strategies this process can be made defined and repeatable.
Language Resources and EvaluationAnCora: Multilevel Annotated Corpora for Catalan and Spanish
291 Citations2008Mariona Taulé, M. Antònia Martí +1 more
AnCora is the largest multilayer annotated corpus of these languages freely available from http://clic.ub.edu/ancora, a multilingual corpus annotated at different linguistic levels consisting of 500,000 words in Catalan and Spanish.
SemEval-2010 Task 13: TempEval-2
286 Citations2010Marc Verhagen, R. Supyan Sauri +2 more
Tempeval-2 comprises evaluation tasks for time expressions, events and temporal relations, the latter of which was split up in four sub tasks, motivated by the notion that smaller subtasks would make both data preparation and temporal relation extraction easier.
The Specification Language TimeML
164 Citations2005James Pustejovsky, Robert Ingria +7 more
This paper provides a description of TimeML, a rich specification language for event and temporal expressions in natural language text, developed in the context of the AQUAINT program on Question Answering Systems, and demonstrates the expressiveness of timeML for a broad range of syntactic and semantic contexts.
TIPSem (English and Spanish): Evaluating CRFs and Semantic Roles in TempEval-2
142 Citations2010Héctor Llorens, Estela Saquete +1 more
TIPSem, a system to extract temporal information from natural language texts for English and Spanish, learns CRF models from training data and achieves the best F1 score in all the tasks.
TRIPS and TRIOS System for TempEval-2: Extracting Temporal Information from Text
115 Citations2010Naushad UzZaman, James F. Allen
This paper describes two systems, TRIPS and TRIOS, that were submitted to the TempEval 2 challenge, for extracting temporal information from raw text using a combination of deep semantic parsing, Markov Logic Networks and Conditional Random Field classifiers.
Language Resources and EvaluationThe TempEval challenge: identifying temporal relations in text
101 Citations2009Marc Verhagen, Robert Gaizauskas +4 more
The TempEval task and the systems that participated in the evaluation are described and how further task decomposition can bring even more structure to the evaluation of temporal relations is described.
arXiv (Cornell University)TempEval-3: Evaluating Events, Time Expressions, and Temporal Relations
85 Citations2012Naushad UzZaman, Héctor Llorens +4 more
The TempEval-3 task is described, incorporating a three-part task structure covering event, temporal expression and temporal relation extraction; a larger dataset; and single overall task quality scores.
International Conference on Semantic Computing (ICSC 2007)Learning by Reading by Learning to Read
55 Citations2007Sergei Nirenburg, Tim Oates +1 more
The architecture of the GIBRALTAR system, which is based on the OntoSem semantic analyzer, is described, which learns by reading by learning to read and extracts knowledge about the topics of the texts and knowledge about language that enables improved text processing.
Temporal Evaluation
55 Citations2011Naushad UzZaman, James F. Allen
A new method is proposed for evaluating systems that extract temporal information from text that uses temporal closure to reward relations that are equivalent but distinct, making comparison between different systems straightforward.
Information Processing & ManagementApplying semantic knowledge to the automatic processing of temporal expressions and events in natural language
35 Citations2012Héctor Llorens, Estela Saquete +1 more
It is concluded that the application of semantic knowledge leads to more general models and aids in the recognition of temporal entities that are ambiguous at shallower language analysis levels and that lexical semantics and semantic roles have complementary advantages, and that it is useful to combine them.
Evaluating Temporal Information Understanding with Temporal Question Answering
16 Citations2012Naushad UzZaman, Héctor Llorens +1 more
The QA-based evaluation results suggest that (i) the available temporal annotations are not complete, and (ii) QA provides a less costly and more reliable way of evaluating temporal understanding systems.
Analysing Temporally Annotated Corpora with CAVaT
16 Citations2010Leon Derczynski, Robert Gaizauskas
The reporting part of CAVaT is presented, and then its error-checking ability, including the workings of several novel TimeML document verification methods are presented, followed by the execution of some example tasks using the tool to show relations between times, events, signals and links.
arXiv (Cornell University)A Corpus-based Study of Temporal Signals
13 Citations2012Leon Derczynski, Robert Gaizauskas
The role of temporal signals in temporal relation extraction is investigated and a quantitative analysis of these expres sions in the TimeBank annotated corpus is provided.
arXiv (Cornell University)TimeML-strict: clarifying temporal annotation
12 Citations2013Leon Derczynski, Héctor Llorens +1 more
This paper introduces TimeML-strict: a valid, unambiguous, and easy-to-process subset of TimeML that is compatible with the laxer TimeML standard.
Merging Temporal Annotations
9 Citations2012Héctor Llorens, Naushad UzZaman +1 more
This work presents automatic algorithms specifically for merging temporal annotations, evaluated merging the annotations of three state-of-the-art systems on the gold standard corpora and the correctness of the merged annotation improved over that of individual annotations and baseline merging algorithms.
Interpreting the temporal aspects of language
5 Citations2012James F. Allen, Naushad UzZaman
A hybrid system to automatically extract temporal information from raw text by extracting events, temporal expressions and identifying temporal relations between entities and a question-answering system that can answer temporal questions with temporal reasoning are presented.