Confidence estimation for information extraction

Published 1 January 2004Open access

Aron Culotta, Andrew McCallum

Citations115

TL;DR

This work evaluates a information extraction system based on a linear-chain conditional random field (CRF), a probabilistic model which has performed well on information extraction tasks because of its ability to capture arbitrary, overlapping features of the input in a Markov model.

Abstract

Information extraction techniques automatically create structured databases from unstructured data sources, such as the Web or newswire documents. Despite the successes of these systems, accuracy will always be imperfect. For many reasons, it is highly desirable to accurately estimate the confidence the system has in the correctness of each extracted field. The information extraction system we evaluate is based on a linear-chain conditional random field (CRF), a probabilistic model which has performed well on information extraction tasks because of its ability to capture arbitrary, overlapping features of the input in a Markov model. We implement several techniques to estimate the confidence of both extracted fields and entire multi-field records, obtaining an average precision of 98% for retrieving correct fields and 87% for multi-field records.

Keywords

Computer Science

ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

12,978 Citations2001John Lafferty, Andrew McCallum +1 more

This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.

View PDF

Shallow parsing with conditional random fields

1,251 Citations2003Fei Sha, Fernando Pereira

This work shows how to train a conditional random field to achieve performance as good as any reported base noun-phrase chunking method on the CoNLL task, and better than any reported single model.

View PDF

Max-Margin Markov Networks

1,251 Citations2003Ben Taskar, Carlos Guestrin +1 more

Maximum margin Markov (M3) networks incorporate both kernels, which efficiently deal with high-dimensional features, and the ability to capture correlations in structured data, and a new theoretical bound for generalization in structured domains is provided.

Lecture notes in computer scienceActive Hidden Markov Models for Information Extraction

525 Citations2001Tobias Scheffer, Christian Decomain +1 more

This paper considers the more challenging task of learning hidden Markov models (HMMs) when only partially (sparsely) labeled documents are available for training, and describes an EM style algorithm for learning HMMs from partially labeled data.

Discriminative Reranking for Natural Language Parsing

443 Citations2000Michael J. Collins

Active Learning for Natural Language Parsing and Information Extraction

290 Citations1999Cynthia A. Thompson, Mary Elaine Califf +1 more

It is shown that active learning can signicantly reduce the number of annotated examples required to achieve a given level of performance for these complex tasks: semantic parsing and information extraction.

CORE Scholar (Wright State University)Interactive information extraction with constrained conditional random fields

142 Citations2004Trausti Kristjansson, Aron Culotta +2 more

This work applies a constrained Viterbi decoding which finds the optimal field assignments consistent with the fields explicitly specified or corrected by the user; and a mechanism for estimating the confidence of each extracted field, so that low-confidence extractions can be highlighted.

View PDF

Probabilistic combination of text classifiers using reliability indicators

59 Citations2002Paul N. Bennett, Susan Dumais +1 more

A probabilistic method for combining classifiers that considers the context-sensitive reliabilities of contributing classifiers, which harnesses reliability indicators---variables that provide a valuable signal about the performance of classifiers in different situations.

ScholarWorks@UMassAmherst (University of Massachusetts Amherst)A Note on the Unification of Information Extraction and Data Mining using Conditional-Probability, Relational Models

56 Citations2003Andrew McCallum, David Jensen

This position paper proposes the use of unified, relational, undirected graphical models for information extraction and data mining, in which extraction decisions and data-mining decisions are made in the same probabilistic “currency” with a common inference procedure— each component thus being able to make up for the weaknesses of the other and therefore improving the performance of both.

View PDF

Confidence estimation for translation prediction

53 Citations2003Simona Gandrabur, George Foster

This work attempts to learn probabilities of correctness for various model predictions, based on the native probabilites given by the original model and on features of the current context, within a statistical machine translation application.

View PDF