Unsupervised Structure Prediction with Non-Parallel Multilingual Guidance
Edinburgh Research Explorer (University of Edinburgh)Published 29 June 2018Open access
Shay B. Cohen, Dipanjan Das, Noah A. Smith
Citations87
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work describes a method for prediction of linguistic structure in a language for which only unlabeled data is available, using annotated data from a set of one or more helper languages, based on a model that locally mixes between supervised models from the helper languages.
Abstract
We describe a method for prediction of linguistic structure in a language for which only unlabeled data is available, using annotated data from a set of one or more helper languages. Our approach is based on a model that locally mixes between supervised models from the helper languages. Parallel data is not used, allowing the technique to be applied even in domains where human-translated texts are unavailable. We obtain state-of-theart performance for two tasks of structure prediction: unsupervised part-of-speech tagging and unsupervised dependency parsing.
Keywords
Computer Science
Mathematical ProgrammingOn the limited memory BFGS method for large scale optimization
8,529 Citations1989Dong C. Liu, Jorge Nocedal
The numerical tests indicate that the L-BFGS method is faster than the method of Buckley and LeNir, and is better able to use additional storage to accelerate convergence, and the convergence properties are studied to prove global convergence on uniformly convex problems.
Building a Large Annotated Corpus of English: The Penn Treebank
7,528 Citations1993Mitchell P. Marcus
CoNLL-X shared task on multilingual dependency parsing
976 Citations2006Sabine Buchholz, Erwin Marsi
How treebanks for 13 languages were converted into the same dependency format and how parsing performance was measured is described and general conclusions about multi-lingual parsing are drawn.
Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt)The CoNLL 2007 Shared Task on Dependency Parsing
662 Citations2007Joakim Nivre, Johan Hall +5 more
Tagging English text with a probabilistic model
478 Citations1994Bernard Mérialdo
ScholarlyCommons (University of Pennsylvania)Posterior Regularization for Structured Latent Variable Models
462 Citations2010Kuzman Ganchev, Joäo Graça +2 more
Corpus-based induction of syntactic structure
453 Citations2004Dan Klein, Christopher D. Manning
This work presents a generative model for the unsupervised learning of dependency structures and describes the multiplicative combination of this dependency model with a model of linear constituency that works and is robust cross-linguistically.
Contrastive estimation
320 Citations2005Noah A. Smith, Jason Eisner
A novel approach, contrastive estimation, is described, which outperforms EM, is more robust to degradations of the dictionary, and can largely recover by modeling additional features.
Empirical Methods in Natural Language ProcessingMulti-Source Transfer of Delexicalized Dependency Parsers
314 Citations2011Ryan McDonald, Slav Petrov +1 more
This work demonstrates that delexicalized parsers can be directly transferred between languages, producing significantly higher accuracies than unsupervised parsers and shows that simple methods for introducing multiple source languages can significantly improve the overall quality of the resulting parsers.
Edinburgh Research Explorer (University of Edinburgh)Bayesian Inference for PCFGs via Markov Chain Monte Carlo
280 Citations2007Mark Johnson, Thomas L. Griffiths +1 more
Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora
247 Citations2001David Yarowsky, Grace Ngai
This paper investigates the potential for projecting linguistic annotations including part-of-speech tags and base noun phrase bracketings from one language to another via automatically word-aligned parallel corpora.
Unsupervised Part-of-Speech Tagging with Bilingual Graph-Based Projections
247 Citations2011Dipanjan Das, Slav Petrov
A novel approach for inducing unsupervised part-of-speech taggers for languages that have no labeled training data, but have translated text in a resource-rich language, using graph-based label propagation for cross-lingual knowledge transfer.
Painless Unsupervised Learning with Features
224 Citations2010Taylor Berg-Kirkpatrick, Alexandre Bouchard‐Côté +2 more
This work shows how features can easily be added to standard generative models for unsupervised learning, without requiring complex new training methods, and applies this technique to part-of-speech induction, grammar induction, word alignment, and word segmentation.
Prototype-driven learning for sequence models
178 Citations2006Aria Haghighi, Dan Klein
This work investigates prototype-driven learning for primarily unsupervised sequence modeling, where prior knowledge is specified declaratively, by providing a few canonical examples of each target annotation label, then propagated across a corpus using distributional similarity features in a log-linear generative model.
Unsupervised Multilingual Learning for Morphological Segmentation
153 Citations2008Benjamin Snyder, Regina Barzilay
A nonparametric Bayesian model is presented that jointly induces morpheme segmentations of each language under consideration and at the same time identifies cross-lingual morphem patterns, or abstract morphemes, of multiple languages.
DSpace@MIT (Massachusetts Institute of Technology)Using Universal Linguistic Knowledge to Guide Grammar Induction
140 Citations2010Tahira Naseem, Harr Chen +2 more
This work presents an approach to grammar induction that utilizes syntactic universals to improve dependency parsing across a range of languages and outperforms state-of-the-art unsupervised methods by a significant margin.
From Baby Steps to Leapfrog: How ``Less is More'' in Unsupervised Dependency Parsing
137 Citations2010Valentin I. Spitkovsky, Hiyan Alshawi +1 more
Three approaches for unsupervised grammar induction that are sensitive to data complexity are presented and apply them to Klein and Manning's Dependency Model with Valence, beating state-of-the-art and generalizing to the Brown corpus.
Does Baum-Welch re-estimation help taggers?
129 Citations1994David Elworthy
Two experiments designed to determine how much manual training information is needed for speech tagging by Hidden Markov Model suggest that initial biasing of either lexical or transition probabilities is essential to achieve a good accuracy and reveal three distinct patterns of Baum-Welch reestimation.
Improving unsupervised dependency parsing with richer contexts and smoothing
128 Citations2009William Headden, Mark Johnson +1 more
This paper introduces basic valence frames and lexical information into an unsupervised dependency grammar inducer and shows how this additional information can be leveraged via smoothing to produce state-of-the-art results.
Shared logistic normal distributions for soft parameter tying in unsupervised grammar induction
124 Citations2009Shay B. Cohen, Noah A. Smith
A family of priors over probabilistic grammar weights is presented, called the shared logistic normal distribution, enabling factored covariance between the probabilities of different derivation events in the probabilism grammar, providing a new way to encode prior knowledge about an unknown grammar.
Text, speech and language technologyBuilding the Italian Syntactic-Semantic Treebank
123 Citations2003Simonetta Montemagni⋄, Francesco Barsotti +15 more
The paper reports on the design and construction of a multi-layered corpus of Italian, annotated at the syntactic and lexico-semantic levels, whose development is supported by dedicated software augmented with an intelligent interface.
Data Archiving and Networked Services (DANS)Fast Unsupervised Incremental Parsing
115 Citations2007Yoav Seginer
This paper describes an incremental parser and an unsupervised learning algorithm for inducing this parser from plain text, which uses a representation for syntactic structure similar to dependency links which is well-suited for incremental parsing.
Two languages are better than one (for syntactic parsing)
101 Citations2008David Burkett, Dan Klein
It is shown that jointly parsing a bitext can substantially improve parse quality on both sides, and the resulting bitext parser outperforms state-of-the-art monolingual parser baselines in a maximum entropy bitext parsing model.
Unsupervised multilingual grammar induction
81 Citations2009Benjamin Snyder, Tahira Naseem +1 more
A generative Bayesian model is formulated which seeks to explain the observed parallel data through a combination of bilingual and monolingual parameters, and loosely binds parallel trees while allowing language-specific syntactic structure.
Bilingual Parsing with Factored Estimation: Using English to Parse Korean
76 Citations2004David A. Smith, Noah A. Smith
This work describes how simple, commonly understood statistical models, such as statistical dependency parsers, probabilistic context-free grammars, and word-to-word translation models can be effectively combined into a unified bilingual parser that jointly searches for the best English parse, Korean parse, andword alignment.
Parser adaptation and projection with quasi-synchronous grammar features
69 Citations2009David A. Smith, Jason Eisner
This work connects two scenarios in structured learning: adapting a parser trained on one corpus to another annotation style, and projecting syntactic annotations from one language to another, and proposes quasi-synchronous grammar (QG) features for these structured learning tasks.
Novel estimation methods for unsupervised discovery of latent structure in natural language text
62 Citations2007Jason Eisner, Noah A. Smith
The novel estimation methods presented are better suited to adaptation for real engineering tasks than the maximum likelihood baseline, and are shown to achieve significant improvements over maximum likelihood estimation and maximum a posteriori estimation, for a state-of-the-art probabilistic model used in dependency grammar induction.
Phylogenetic Grammar Induction
60 Citations2010Taylor Berg-Kirkpatrick, Dan Klein
Joint induction in the multilingual model substantially outperforms independent learning, with larger gains both from more articulated phylogenies and as well as from increasing numbers of languages.
ScholarlyCommons (University of Pennsylvania)Sparsity in Dependency Grammar Induction
55 Citations2010Jennifer Gillenwater, Kuzman Ganchev +3 more
This work investigates sparsity-inducing penalties on the posterior distributions of parent-child POS tag pairs in the posterior regularization (PR) framework of Graca et al. (2007) and shows that its approach improves on several other state-of-the-art techniques.
Journal of Artificial Intelligence ResearchMultilingual Part-of-Speech Tagging: Two Unsupervised Approaches
54 Citations2009Tahira Naseem, Bethany Snyder +2 more
This work considers two ways of applying this intuition to the problem of unsupervised part-of-speech tagging: a model that directly merges tag structures for a pair of languages into a single sequence and a second model which instead incorporates multilingual context using latent variables.
Edinburgh Research Explorer (University of Edinburgh)Joint Morphological and Syntactic Disambiguation
48 Citations2007Shay B. Cohen, Noah A. Smith
Several efficient, probabilisticallyinterpretable ways to apply joint inference to morphological and syntactic disambiguation using lattice parsing are described and shown to compare favorably to pipeline parsing methods across a variety of component models.
A backoff model for bootstrapping resources for non-English languages
39 Citations2005Chenhai Xi, Rebecca Hwa
This paper proposes a novel approach of combining a bootstrapped resource with a small amount of manually annotated data and shows that this approach achieves a significant improvement over EM and self-training and systems that are only trained on manual annotations.
Journal of Machine Learning ResearchCovariance in Unsupervised Learning of Probabilistic Grammars
28 Citations2010Shay B. Cohen, Noah A. Smith
An alternative to the Dirichlet prior is suggested, a family of logistic normal distributions that permits soft parameter tying within grammars and across Grammars for text in different languages, and empirical gains in a novel learning setting using bilingual, non-parallel data are shown.