Building a semantically annotated corpus of clinical texts
Journal of Biomedical InformaticsPublished 24 January 2009
Angus Roberts, Robert Gaizauskas, Mark Hepple, George Demetriou, Yikun Guo, Ian Roberts
Citations166
SJR quartileQ1
SJR score1.26
SNIP1.62
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The construction of a semantically annotated corpus of clinical texts for use in the development and evaluation of systems for automatically extracting clinically significant information from the textual component of patient records is described.
Abstract
In this paper, we describe the construction of a semantically annotated corpus of clinical texts for use in the development and evaluation of systems for automatically extracting clinically significant information from the textual component of patient records. The paper details the sampling of textual material from a collection of 20,000 cancer patient records, the development of a semantic annotation scheme, the annotation methodology, the distribution of annotations in the final corpus, and the use of the corpus for development of an adaptive information extraction system. The resulting corpus is the most richly semantically annotated resource for clinical text processing built to date, whose value has been demonstrated through its use in developing an effective information extraction system. The detailed presentation of our corpus construction and annotation methodology will be of value to others seeking to build high-quality semantically annotated corpora in biomedical domains.
Keywords
Computer ScienceBiochemistry, Genetics and Molecular Biology
BioinformaticsGENIA corpus—a semantically annotated corpus for bio-textmining
1,243 Citations2003JD Kim, Tomoko Ohta +2 more
Methods of Information in MedicineThe Unified Medical Language System
1,187 Citations1993Betsy L. Humphreys, Alexa T. McCray +1 more
The UMLS project and current developments in high-speed, high-capacity international networks are converging in ways that have great potential for enhancing access to biomedical information.
Journal of the American Medical Informatics AssociationAgreement, the F-Measure, and Reliability in Information Retrieval
938 Citations2005George Hripcsak
It can be shown that the average F-measure among pairs of experts is numerically identical to the average positive specific agreement among experts and that kappa approaches these measures as the number of negative cases grows large.
International Journal of Molecular SciencesTimeML: Robust Specification of Event and Temporal Expressions in Text
717 Citations2003James Pustejovsky, José M. Castaño +6 more
This review focuses on the regulatory role of CEACAM1 in liver and biliary tract architecture in health and disease, and on its metabolic role and function as an immune checkpoint regulator of hepatic inflammation.
Automatically generating extraction patterns from untagged text
620 Citations1996Ellen Riloff
This work has developed a system called AutoSlog-TS that creates dictionaries of extraction patterns using only untagged text, and in experiments with the MUG-4 terrorism domain, created a dictionary of extraction pattern that performed comparably to a dictionary created by autoSlog, using only preclassified texts as input.
BMC BioinformaticsCorpus annotation for mining biomedical events from literature
396 Citations2008Jin-Dong Kim, Tomoko Ohta +1 more
A new type of semantic annotation, event annotation, is completed, which is an addition to the existing annotations in the GENIA corpus, and is expected to become a valuable resource for NLP (Natural Language Processing)-based TM in the bio-medical domain.
GATE
385 Citations2001Hamish Cunningham, Diana Maynard +2 more
GATE, a framework and graphical development environment which enables users to develop and deploy language engineering components and resources in a robust fashion, and can be used to develop applications and resources in multiple languages, based on its thorough Unicode support.
A shared task involving multi-label classification of clinical free text
359 Citations2007John Pestian, C. Brew +5 more
A shared task involving the assignment of ICD-9-CM codes to radiology reports resulted in the first freely distributable corpus of fully anonymized clinical text, suggesting that human-like performance on this task is within the reach of currently available technologies.
Robust temporal processing of news
358 Citations2000Inderjeet Mani, George D. Wilson
An annotation scheme for temporal expressions, and a method for resolving temporal expressions in print and broadcast news, based on both hand-crafted and machine-learnt rules are described.
SemEval-2007 task 15
291 Citations2007Marc Verhagen, Robert Gaizauskas +4 more
The TempEval task proposes a simple way to evaluate automatic extraction of temporal relations by defining three sub tasks that allow pairwise evaluation of temporal Relations.
Active Learning for Natural Language Parsing and Information Extraction
290 Citations1999Cynthia A. Thompson, Mary Elaine Califf +1 more
It is shown that active learning can signicantly reduce the number of annotated examples required to achieve a given level of performance for these complex tasks: semantic parsing and information extraction.
Classifying semantic relations in bioscience texts
259 Citations2004Barbara Rosario, Marti A. Hearst
This work examines the problem of distinguishing among seven relation types that can occur between the entities "treatment" and "disease" in bioscience text, and finds that the latter help achieve high classification accuracy.
BMC BioinformaticsGENETAG: a tagged corpus for gene/protein named entity recognition
231 Citations2005Lorraine Tanabe, Natalie Xie +3 more
The annotation of GENETAG required intricate manual judgments by annotators which hindered tagging consistency, and the data were pre-segmented into words, to provide indices supporting comparison of system responses to the "gold standard", however, character- based indices would have been more robust than word-based indices.
Journal of Biomedical InformaticsNatural language processing to extract medical problems from electronic clinical documents: Performance evaluation
203 Citations2005Stéphane Meystre, Peter J. Haug
The NLP part of this system uses the UMLS MetaMap Transfer (MMTx) application and a negation detection algorithm called NegEx to extract 80 different medical problems selected for their frequency of use in the authors' institution.
North American Chapter of the Association for Computational LinguisticsIntegrated Annotation for Biomedical Information Extraction
184 Citations2004Seth Kulick, Ann Bies +8 more
An approach to two areas of biomedical information extraction, drug development and cancer genomics using a framework which includes corpus annotation integrated at multiple levels: a Treebank containing syntactic structure, a Propbank containing predicate-argument structure, and annotation of entities and relations among the entities.
BMC Medical Informatics and Decision MakingA controlled trial of automated classification of negation from clinical notes
143 Citations2005Peter L. Elkin, Steven H. Brown +5 more
International Journal of Medical InformaticsProtein names and how to find them
127 Citations2002Kristofer Franzén, Gunnar Eriksson +4 more
This work describes how Yapex, a system for the automatic identification of protein names in text, was implemented and evaluated under four different notions of correctness and compares its performance to that of another publicly available system for protein name recognition.
Discourse chunking and its application to sentence compression
122 Citations2005Caroline Sporleder, Mirella Lapata
This paper introduces discourse chunking (i.e., the identification of intra-sentential nucleus and satellite spans) as an alternative to full-scale discourse parsing as well as exploiting knowledge-lean features and small amounts of discourse annotations.
Journal of the American Medical Informatics Association"Understanding" Medical School Curriculum Content Using KnowledgeMap
121 Citations2003Joshua C. Denny
The prototypic KM system provided an encouraging rate of concept extraction for representative medical curricular texts and should be evaluated for their ability to allow administrators, lecturers, and students to navigate through the medical curriculum to locate redundancies, find interrelated information, and identify omissions.
Lecture notes in computer scienceOverview of the ImageCLEFmed 2006 Medical Retrieval and Medical Annotation Tasks
112 Citations2007Henning Müller, Thomas Deselaers +4 more
The medical automatic annotation task used a larger database of 10,000 training images from 116 classes, up from 9,000 images from 57 classes in 2005, which demonstrates a clear improvement in performance.
PubMedThe CLEF corpus: semantic annotation of clinical text.
107 Citations2007Angus Roberts, Robert Gaizauskas +9 more
An annotation methodology is described and encouraging initial results of inter-annotator agreement are reported, and Comparisons are made between different text sub-genres, and between annotators with different skills.
Methods of Information in MedicineEvaluating Natural Language Processors in the Clinical Domain
105 Citations1998George Hripcsak, C Friedman
This paper describes a set of criteria aimed at improving the quality of NLP evaluation studies and discusses a number of factors which contribute to the complexity that is inherent in the task of evaluating natural language systems.
Lecture notes in computer scienceSVM Based Learning System for Information Extraction
104 Citations2005Yaoyong Li, Kalina Bontcheva +1 more
One distinctive feature of this SVM-based learning system for information extraction is the use of a variant of the S VM, the SVM with uneven margins, which is particularly helpful for small training datasets.
Multi-way relation classification
81 Citations2005Barbara Rosario, Marti A. Hearst
This work uses two graphical models and a neural net for the classification of the interactions between proteins in bioscience text, achieving an accuracy of 64% for a 10-way distinction between relation types.
Overview of the ImageCLEFmed 2006 medical retrieval and annotation tasks
81 Citations2006Henning Müller, Thomas Deselaers +4 more
This paper describes the medial image retrieval and the medical annotation tasks of ImageCLEF 2006, with respect to the goals, databases used, topics created and distributed among participants, results and techniques used.
Journal of the American Medical Informatics AssociationAdvancing Biomedical Image Retrieval: Development and Analysis of a Test Collection
73 Citations2006William Hersh, Henning Müller +4 more
Extracting clinical relationships from patient narratives
51 Citations2008Angus Roberts, Robert Gaizauskas +1 more
A supervised machine learning system is described, trained with a corpus of oncology narratives hand-annotated with clinically important relationships, and various shallow features are extracted from these texts, and used to train statistical classifiers.
BMC BioinformaticsMining clinical relationships from patient narratives
42 Citations2008Angus Roberts, Robert Gaizauskas +2 more
It is shown that it is possible to extract important clinical relationships from text, using supervised statistical ML techniques, at levels of accuracy approaching those of human annotators.
Combining Terminology Resources and Statistical Methods for Entity Recognition: an Evaluation
37 Citations2008Angus Roberts, Robert Gaizauskas +2 more
This work combines lexical lookup with simple filtering of ambiguous terms, to improve precision, and shows that the combined method boosts precision with little loss of recall, and that linkage from recognised entities back to the domain knowledge resources can be maintained.
CLEF: joining up healthcare with clinical and post-genomic research
31 Citations2003A Rector, Jamie Rogers +9 more
The overall information flow and technical approach designed to meet these aims within a Grid framework are described and details of work on language technology and ethical issues are discussed in separate papers at this conference.
Open Research Online (The Open University)Summarisation and Visualisation of e-Health Data Repositories
28 Citations2005Catalina Hallett, Richard Power +1 more
A system to perform the task of generating a diverse range of textual and graphical summaries of a patient’s clinical history from a data-encoded model, a chronicle, representing the record of the patient's medical history is described.
PubMedBuilding and evaluating annotated corpora for medical NLP systems.
19 Citations2006Philip V. Ogren, Guergana Savova +2 more
Results from a pilot annotation project in which four retrieval experts annotated the same clinical notes so that four-way inter-annotator agreement could be calculated are encouraging to scale up the project using the same guidelines and annotation schema.
The ITI TXM Corpora: Tissue Expressions and Protein-Protein Interactions
18 Citations2008Beatrice Alex, Claire Grover +7 more
The selected markables and the annotation process of the ITI TXM corpora are described, and a detailed breakdown of the inter-annotator agreement (IAA) is provided.
The CLEF Chronicle: Patient Histories Derived from Electronic Health Records
17 Citations2006Jamie Rogers, Colin Puleston +1 more
The reconstruction of patient chronicles from clinical records and the subsequent definition and execution of sophisticated clinical queries across populations of chronicles are described.
A Large-Scale Resource for Storing and Recognizing Technical Terminology.
5 Citations2004Henk Harkema, Robert Gaizauskas +5 more
The design and implementation of Termino, a large-scale terminological resource for text processing, is discussed, a large-scale relational database for storing terminological information and compiling state machines from this database to do term recognition.
Language Resources and EvaluationANNALIST - ANNotation ALIgnment and Scoring Tool
4 Citations2008George Demetriou, Robert Gaizauskas +2 more
The paper reports on results from using ANNALIST in real-world situations in comparison to other scorers which are more established in the literature.