Confidence Estimation for Knowledge Base Population
Published 1 September 2013
Xiang Li, Ralph Grishman
Citations5
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A confidence estimation model based on the Maximum Entropy framework is proposed, obtaining an average precision of 83.5%, Pearson coefficient of 54.2%, and 2.3%absolute improvement in F-measure score through a weighted voting strategy.
Abstract
Information extraction systems automati-cally extract structured information from machine-readable documents, such as newswire, web, and multimedia. Despite significant improvement, the performance is far from perfect. Hence, it is useful to accurately estimate confidence in the cor-rectness of the extracted information. Us-ing the Knowledge Base Population Slot Filling task as a case study, we propose a confidence estimation model based on the Maximum Entropy framework, obtaining an average precision of 83.5%, Pearson coefficient of 54.2%, and 2.3 % absolute improvement in F-measure score through a weighted voting strategy. 1
Keywords
Computer ScienceDecision Sciences
Computational LinguisticsA maximum entropy approach to natural language processing
3,120 Citations1996Adam Berger, Vincent J. Della Pietra +1 more
A maximum-likelihood approach for automatically constructing maximum entropy models is presented and how to implement this approach efficiently is described, using as examples several problems in natural language processing.
Lecture notes in computer scienceActive Hidden Markov Models for Information Extraction
525 Citations2001Tobias Scheffer, Christian Decomain +1 more
This paper considers the more challenging task of learning hidden Markov models (HMMs) when only partially (sparsely) labeled documents are available for training, and describes an EM style algorithm for learning HMMs from partially labeled data.
Overview of the TAC 2010 Knowledge Base Population Track
419 Citations2010Heng Ji, Ralph Grishman +3 more
An overview of the task definition and annotation challenges associated with KBP2010 is provided and the evaluation results and lessons that are learned are discussed based on detailed analysis.
Knowledge Base Population: Successful Approaches and Challenges
311 Citations2011Heng Ji, Ralph Grishman
The techniques which can serve as a basis for a good KBP system are provided, the remaining challenges by comparison with traditional Information Extraction (IE) and Question Answering (QA) tasks are laid out, and some suggestions to address these challenges are provided.
Active Learning for Natural Language Parsing and Information Extraction
290 Citations1999Cynthia A. Thompson, Mary Elaine Califf +1 more
It is shown that active learning can signicantly reduce the number of annotated examples required to achieve a given level of performance for these complex tasks: semantic parsing and information extraction.
Confidence estimation for information extraction
115 Citations2004Aron Culotta, Andrew McCallum
This work evaluates a information extraction system based on a linear-chain conditional random field (CRF), a probabilistic model which has performed well on information extraction tasks because of its ability to capture arbitrary, overlapping features of the input in a Markov model.
Goodness: A Method for Measuring Machine Translation Confidence
80 Citations2011Nguyễn Bách, Fei Huang +1 more
A novel framework to predict word-level and sentence-level MT errors with a large number of novel features and a visualization prototype of MT errors at the word and sentence levels with the objective to improve post-editor productivity.
Theory and applications of categoriesCUNY-BLENDER TAC-KBP2010 Entity Linking and Slot Filling System Description
56 Citations2010Zheng Chen, Suzanne Tamang +7 more
In the KBP2010 evaluation, the CUNY-BLENDER entity linking system explored the slot filling attributes that may potentially help disambiguate entity mentions and results show that this feedback approach can achieve 9.1% absolute improvement on micro-average accuracy over the baseline using vector space model.
Confidence estimation for translation prediction
53 Citations2003Simona Gandrabur, George Foster
This work attempts to learn probabilities of correctness for various model predictions, based on the native probabilites given by the original model and on features of the current context, within a statistical machine translation application.
Maximum Entropy Confidence Estimation for Speech Recognition
37 Citations2007Christopher White, Jasha Droppo +2 more
This paper replaces the standard maximum likelihood classifier with a maximum entropy framework, which carries the dual advantages discriminative training and reasonable generalization, and includes a number of alternative features that provide consistent gains for all of the test sets.
New York University 2011 system for KBP slot filling
34 Citations2011Ang Sun, Ralph Grishman +2 more
The NYU 2012 SF system has a similar architecture to the NYU 2011 system and improved the distant-supervision based slot-filling component with a few techniques including filtering errors by statistical measures collected from the source corpus, and relabeling erroneous training examples with a set of maximum entropy models and by applying bootstrapped/hand-coded patterns.
New York University 2012 System for KBP Slot Filling
23 Citations2012Bonan Min, Xiang Li +2 more
Performance confidence estimation for automatic summarization
19 Citations2009Annie Louis, Ani Nenkova
This work addresses the task of automatically predicting if summarization system performance will be good or bad based on features derived directly from either single- or multi-document inputs, and examines different formulations of the classification task which considerably influence performance.
SIAM International Conference on Data MiningConfidence Estimation Methods for Partially Supervised Information Extraction.
19 Citations2006Eugene Agichtein
Hidden Markov Models for Information Extraction
10 Citations2001Nancy R. Zhang
This paper uses a machine learning approach to examine the effectiveness of HMMs on extracting information of varying levels of structure and a stochastic optimization procedure is used to find the optimal structure for a given task.