Learning from measurements in exponential families
Published 14 June 2009
Percy Liang, Michael I. Jordan, Dan Klein
Citations114
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A Bayesian decision-theoretic framework is presented, which allows us to both integrate diverse measurements and choose new measurements to make, and a variational inference algorithm is used, which exploits exponential family duality.
Abstract
Given a model family and a set of unlabeled examples, one could either label specific examples or state general constraints---both provide information about the desired model. In general, what is the most cost-effective way to learn? To address this question, we introduce measurements, a general class of mechanisms for providing information about a target model. We present a Bayesian decision-theoretic framework, which allows us to both integrate diverse measurements and choose new measurements to make. We use a variational inference algorithm, which exploits exponential family duality. The merits of our approach are demonstrated on two sequence labeling tasks.
Keywords
Computer Science
ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
12,978 Citations2001John Lafferty, Andrew McCallum +1 more
This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
Class-based n -gram models of natural language
2,893 Citations1992Peter F. Brown, P.V. deSouza +3 more
This work addresses the problem of predicting a word from previous words in a sample of text and discusses n-gram models based on classes of words, finding that these models are able to extract classes that have the flavor of either syntactically based groupings or semanticallybased groupings, depending on the nature of the underlying statistics.
Statistical ScienceBayesian Experimental Design: A Review
1,866 Citations1995Kathryn Chaloner, Isabella Verdinelli
This paper reviews the literature on Bayesian experimental design, both for linear and nonlinear models, and presents a uniied view of the topic by putting experimental design in a decision theoretic framework.
Toward Optimal Active Learning through Sampling Estimation of Error Reduction
824 Citations2001Nicholas Roy, Andrew McCallum
CMS books in mathematicsTechniques of Variational Analysis
551 Citations2005Jonathan M. Borwein, Qiji Jim Zhu
The techniques of variational analysis is universally compatible with any devices to read and is available in the digital library an online access to it is set as public so you can download it instantly.
Learning from labeled features using generalized expectation criteria
254 Citations2008Gregory Druck, Gideon Mann +1 more
This paper proposes a method for training discriminative probabilistic models with labeled features and unlabeled instances and expresses soft constraints using generalized expectation (GE) criteria terms in a parameter estimation objective function that express preferences on values of a model expectation.
Journal of Machine Learning ResearchMaximum Entropy Density Estimation with Generalized Regularization and an Application to Species Distribution Modeling
206 Citations2007Miroslav Dudı́k, Steven J. Phillips +1 more
This work proposes an algorithm solving a large and general subclass of generalized maximum entropy problems, including all discussed in the paper, and proves its convergence.
Guiding Semi-Supervision with Constraint-Driven Learning
195 Citations2007Ming‐Wei Chang, Lev Ratinov +1 more
The experimental results presented in the information extraction domain demonstrate that applying constraints helps the model to generate better feedback during learning, and hence the framework allows for high performance learning with significantly less training data than was possible before on these tasks.
Prototype-driven learning for sequence models
178 Citations2006Aria Haghighi, Dan Klein
This work investigates prototype-driven learning for primarily unsupervised sequence modeling, where prior knowledge is specified declaratively, by providing a few canonical examples of each target annotation label, then propagated across a corpus using distributional similarity features in a log-linear generative model.
Simple, robust, scalable semi-supervised learning via expectation regularization
162 Citations2007Gideon Mann, Andrew McCallum
Expectation regularization is presented, a semi-supervised learning method for exponential family parametric models that augments the traditional conditional label-likelihood objective function with an additional term that encourages model predictions on unlabeled data to match certain expectations---such as label priors.
Expectation Maximization and Posterior Constraints
151 Citations2007Kuzman Ganchev, Ben Taskar +1 more
This paper presents an efficient, principled way to inject rich constraints on the posteriors of latent variables into the EM algorithm, and shows that simple, intuitive posterior constraints can greatly improve the performance over standard baselines and be competitive with more complex, intractable models.
CORE Scholar (Wright State University)Generalized Expectation Criteria for Semi-Supervised Learning of Conditional Random Fields
130 Citations2008Gideon Mann, Andrew McCallum
This paper presents a semi-supervised training method for linear-chain conditional random fields that makes use of labeled features rather than labeled instances by using generalized expectation criteria to express a preference for parameter settings in which the model’s distribution on unlabeled data matches a target distribution.
Compressed sensing and Bayesian experimental design
116 Citations2008Matthias Seeger, Hannes Nickisch
This work relates compressed sensing with Bayesian experimental design and provides a novel efficient approximate method for the latter, based on expectation propagation, which is the first successful attempt at "learning compressed sensing" for images of realistic size.
Estimating labels from label proportions
105 Citations2008Novi Quadrianto, Alex Smola +2 more
This work presents consistent estimators which can reconstruct the correct labels with high probability in a uniform convergence sense in areas like e-commerce, spam filtering and improper content detection.