Semi-supervised conditional random fields for improved sequence segmentation and labeling
Published 1 January 2006Open access
Feng Jiao, Shaojun Wang, Chi‐Hoon Lee, Russell Greiner, Dale Schuurmans
Citations130
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new semi-supervised training procedure for conditional random fields (CRFs) that can be used to train sequence segmentors and labelers from a combination of labeled and unlabeled training data is presented, based on extending the minimum entropy regularization framework to the structured prediction case.
Abstract
We present a new semi-supervised training procedure for conditional random fields (CRFs) that can be used to train sequence segmentors and labelers from a combination of labeled and unlabeled training data. Our approach is based on extending the minimum entropy regularization framework to the structured prediction case, yielding a training objective that combines unlabeled conditional entropy with labeled conditional likelihood. Although the training objective is no longer concave, it can still be used to improve an initial model (e.g. obtained from supervised training) by iterative ascent. We apply our new training algorithm to the problem of identifying gene and protein mentions in biological texts, and show that incorporating unlabeled data improves the performance of the supervised CRF in this case.
Keywords
Computer ScienceBiochemistry, Genetics and Molecular Biology
Elements of Information Theory
37,533 Citations2001Thomas M. Cover, Joy A. Thomas
Cambridge University Press eBooksConvex Optimization
31,266 Citations2004Stephen Boyd, Lieven Vandenberghe
ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
12,978 Citations2001John Lafferty, Andrew McCallum +1 more
This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
Pattern classification and scene analysis
12,643 Citations1973Richard O. Duda, Peter E. Hart
Numerical Optimization
9,247 Citations1999Jorge Nocedal, Stephen J. Wright
Combining labeled and unlabeled data with co-training
5,604 Citations1998Avrim Blum, Tom M. Mitchell
MPG.PuRe (Max Planck Society)Learning with Local and Global Consistency
3,750 Citations2003Dengyong Zhou, Olivier Bousquet +3 more
A principled approach to semi-supervised learning is to design a classifying function which is sufficiently smooth with respect to the intrinsic structure collectively revealed by known labeled and unlabeled points.
Semi-supervised learning using Gaussian fields and harmonic functions
3,377 Citations2003Xiaojin Zhu, Zoubin Ghahramani +1 more
Machine LearningText Classification from Labeled and Unlabeled Documents using EM
2,749 Citations2000Kamal Nigam, Andrew Kachites McCallum +2 more
This paper shows that the accuracy of learned text classifiers can be improved by augmenting a small number of labeled training documents with a large pool of unlabeled documents, and presents two extensions to the algorithm that improve classification accuracy under these conditions.
Unsupervised word sense disambiguation rivaling supervised methods
2,408 Citations1995David Yarowsky
An unsupervised learning algorithm for sense disambiguation that, when trained on unannotated English text, rivals the performance of supervised techniques that require time-consuming hand annotations.
Semi-supervised Learning by Entropy Minimization.
1,021 Citations2005Yves Grandvalet, Yoshua Bengio
This framework, which motivates minimum entropy regularization, enables to incorporate unlabeled data in the standard supervised learning, and includes other approaches to the semi-supervised problem as particular or limiting cases.
Computational Statistics & Data AnalysisA classification EM algorithm for clustering and two stochastic versions
750 Citations1992Gilles Celeux, Gérard Govaert
Two stochastic algorithms are derived from this general Classification EM algorithm, incorporating random perturbations, to reduce the initial-position dependence of the classical optimization clustering algorithms.
Learning from labeled and unlabeled data on a directed graph
403 Citations2005Dengyong Zhou, Jiayuan Huang +1 more
A general framework for learning from labeled and unlabeled data on a directed graph in which the structure of the graph including the directionality of the edges is considered, which generalizes the spectral clustering approach for undirected graphs.
Accelerated training of conditional random fields with stochastic gradient methods
284 Citations2006S. V. N. Vishwanathan, Nicol N. Schraudolph +2 more
Stochastic Meta-Descent (SMD), a stochastic gradient optimization method with gain vector adaptation, is applied to the training of Conditional Random Fields (CRFs) and the resulting optimizer converges to the same quality of solution over an order of magnitude faster than limited-memory BFGS.
BMC BioinformaticsIdentifying gene and protein mentions in text using conditional random fields
223 Citations2005Ryan McDonald, Fernando Pereira
A diverse feature set containing standard orthographic features combined with expert features in the form of gene and biological term lexicons is employed to achieve a precision of 86.4% and recall of 78.7% for tagging gene and protein mentions from text using the probabilistic sequence tagging framework of conditional random fields.
IEEE Transactions on Information TheoryThe relative value of labeled and unlabeled samples in pattern recognition with an unknown mixing parameter
206 Citations1996V. Castelli, Thomas M. Cover
It is shown that labeled samples are necessary to construct a classification rule and that they are exponentially more valuable than unlabeled samples.
Computational LinguisticsUnderstanding the Yarowsky Algorithm
132 Citations2004Steven Abney
A number of variants of the Yarowsky algorithm (though not the original algorithm itself) are shown to optimize either likelihood or a closely related objective function K.
Maximum Margin Semi-Supervised Learning for Structured Variables
99 Citations2005Yasemin Altün, David McAllester +1 more
This paper presents a discriminative approach that utilizes the intrinsic geometry of input patterns revealed by unlabeled data points and derives a maximum-margin formulation of semi-supervised learning for structured variables.
Pattern RecognitionMaximum certainty data partitioning
71 Citations2000Stephen Roberts, Richard Everson +1 more
It is shown that minimisation of partition entropy can be used to estimate the number and structure of probable data generators and the notion of cluster analysis in information-theoretic terms is re-considered.
Semi-supervised sequence modeling with syntactic topic models
52 Citations2005Wei Li, Andrew McCallum
This paper presents an approach that leverages recent work in manifold-learning on sequences to discover word clusters from language data, including both syntactic classes and semantic topics, with statistically-significant improvements over a related semi-supervised sequence tagging method.