A
Published 1 January 2019Open access
Chen Lin, Timothy A. Miller, Dmitriy Dligach, Steven Bethard, Guergana Savova
Citations78
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This study establishes a new state-of-the-art result for the task of CONTAINS temporal relation extraction, by fine-tuning BERT and pre-training domain-specific BERT models on sentence-agnostic temporal relation instances with WordPiece-compatible encodings, and augmenting the labeled data with automatically generated “silver” instances.
Abstract
Classic methods for clinical temporal relation extraction focus on relational candidates within a sentence. On the other hand, break-through Bidirectional Encoder Representations from Transformers (BERT) are trained on large quantities of arbitrary spans of contiguous text instead of sentences. In this study, we aim to build a sentence-agnostic framework for the task of CONTAINS temporal relation extraction. We establish a new state-of-the-art result for the task, 0.684F for in-domain (0.055-point improvement) and 0.565F for cross-domain (0.018-point improvement), by fine-tuning BERT and pre-training domain-specific BERT models on sentence-agnostic temporal relation instances with WordPiece-compatible encodings, and augmenting the labeled data with automatically generated “silver” instances.
Keywords
Computer ScienceBiochemistry, Genetics and Molecular Biology
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
Scientific DataMIMIC-III, a freely accessible critical care database
8,137 Citations2016Alistair E. W. Johnson, Tom Pollard +8 more
BioinformaticsBioBERT: a pre-trained biomedical language representation model for biomedical text mining
7,029 Citations2019Jinhyuk Lee, Wonjin Yoon +5 more
This article introduces BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora that largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre- trained on biomedical Corpora.
International Conference on Computational LinguisticsContextual String Embeddings for Sequence Labeling
1,004 Citations2018Alan Akbik, Duncan A. J. Blythe +1 more
This paper proposes to leverage the internal states of a trained character language model to produce a novel type of word embedding which they refer to as contextual string embeddings, which are fundamentally model words as sequences of characters and are contextualized by their surrounding text.
International Journal of Molecular SciencesTimeML: Robust Specification of Event and Temporal Expressions in Text
717 Citations2003James Pustejovsky, José M. Castaño +6 more
This review focuses on the regulatory role of CEACAM1 in liver and biliary tract architecture in health and disease, and on its metabolic role and function as an immune checkpoint regulator of hepatic inflammation.
Journal of the American Medical Informatics AssociationEvaluating temporal relations in clinical text: 2012 i2b2 Challenge
476 Citations2013Weiyi Sun, Anna Rumshisky +1 more
A corpus of discharge summaries annotated with temporal information was provided to be used for the development and evaluation of temporal reasoning systems, and the best systems overwhelmingly adopted a rule based approach for value normalization.
Annals of NeurologyWhite matter lesion progression, brain atrophy, and cognitive decline: The Austrian stroke prevention study
409 Citations2005Reinhold Schmidt, Stefan Ropele +6 more
SemEval-2016 Task 12: Clinical TempEval
184 Citations2016Steven Bethard, Guergana Savova +4 more
There was a gap between the best systems and human performance, but the gap was less than half the gap of Clinical TempEval 2015.
SemEval-2015 Task 6: Clinical TempEval
173 Citations2015Steven Bethard, Leon Derczynski +3 more
Clinical TempEval 2015 brought the temporal information extraction tasks of past TempEeval campaigns to the clinical domain, with the best systems achieving near-human performance on identifying events and times, but with a large performance gap still remaining for temporal relations.
Journal of Biomedical InformaticsTemporal reasoning with medical data—A review with emphasis on medical natural language processing
157 Citations2007Li Zhou, George Hripcsak
Broad areas for future research include enhancing functions of current MLP systems on processing temporal information, incorporating medical knowledge into temporal reasoning systems, resolving coreference, integrating narrative data with structured data and evaluating these systems.
SemEval-2017 Task 12: Clinical TempEval
122 Citations2017Steven Bethard, Guergana Savova +2 more
Nine sub-tasks were included, covering problems in time expression identification, event expression identification and temporal relation identification, and most tasks observed about a 20 point drop over Clinical TempEval 2016.
Neural Temporal Relation Extraction
108 Citations2017Dmitriy Dligach, Timothy A. Miller +3 more
It is found that neural models with only tokens as input outperform state-of-the-art hand-engineered feature-based models, that convolutional neural networks outperform LSTM models, and that encoding relation arguments with XML tags outperforms a traditional position-based encoding.
Neural Architecture for Temporal Relation Extraction: A Bi-LSTM Approach for Detecting Narrative Containers
100 Citations2017Julien Tourille, Olivier Ferret +2 more
A neural architecture for containment relation identification between medical events and/or temporal expressions is presented and the model achieves an F-measure of 0.613 and outperforms the best result reported on this corpus to date.
Methods of Information in MedicineExtensions to the Time-Oriented Database Model to Support Temporal Reasoning in Medical Expert Systems
66 Citations1991Michael G. Kahn, Lawrence M. Fagan +1 more
Although they do not solve all temporal reasoning problems found in medical decision making, these new structures enable patient database systems to encode complex temporal relationships, to store and retrieve patient data based on multiple clinical contexts and, in ETNET, to modify the reasoning methods available to an expert system based on the onset or conclusion of specific clinical events.
Increasing Informativeness in Temporal Annotation
65 Citations2011James Pustejovsky, Amber Stubbs
A document level information structure the authors call a narrative container (NC) is introduced, designed to increase informativeness and accuracy of temporal relation identification, in the creation of a new temporal annotation over TimeBank.
Structured Learning for Temporal Relation Extraction from Clinical Records
64 Citations2017Artuur Leeuwenberg, Marie‐Francine Moens
This study employs a structured perceptron together with integer linear programming constraints for document-level inference during training and prediction to exploit relational properties of temporality, together with global learning of the relations at the document level.
Representations of Time Expressions for Temporal Relation Extraction with Convolutional Neural Networks
44 Citations2017Chen Lin, Tim Miller +3 more
A method for representing time expressions with single pseudo-tokens for CNNs with a new state-of-the-art result for a clinical temporal relation extraction task is described.
PubMedA comparison of the temporal expressiveness of three database query methods.
32 Citations1995Amar K. Das, Mark A. Musen
The results demonstrate that the T-HELPER system requires simple temporal operations for most protocol-based queries, and TLSQL is the only query method that is sufficiently expressive for the temporal conditions in this system.
Self-training improves Recurrent Neural Networks performance for Temporal Relation Extraction
29 Citations2018Chen Lin, Timothy M. Miller +4 more
This work proposes to build a recurrent neural network with multiple semantically heterogeneous embeddings within a self-training framework that makes use of labeled, unlabeled, and social media data, operates on basic features, and is scalable and generalizable.
Investigating the Challenges of Temporal Relation Extraction from Clinical Text
19 Citations2018Diana Galván-Sosa, Naoaki Okazaki +2 more
To adapt the tree-based LSTM-RNN model proposed by Miwa and Bansal (2016) to temporal relation extraction from clinical text, obtaining a five point improvement over the best 2016 Clinical TempEval system and two points over the state-of-the-art.