Bayesian unsupervised word segmentation with nested Pitman-Yor language modeling
Published 1 January 2009Open access
Daichi Mochihashi, Takeshi Yamada, Naonori Ueda
Citations242
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new Bayesian model for fully unsupervised word segmentation and an efficient blocked Gibbs sampler combined with dynamic programming for inference is proposed and confirmed that it significantly outperforms previous reported results in both phonetic transcripts and standard datasets for Chinese and Japaneseword segmentation.
Abstract
In this paper, we propose a new Bayesian model for fully unsupervised word segmentation and an efficient blocked Gibbs sampler combined with dynamic programming for inference. Our model is a nested hierarchical Pitman-Yor language model, where Pitman-Yor spelling model is embedded in the word model. We confirmed that it significantly outperforms previous reported results in both phonetic transcripts and standard datasets for Chinese and Japanese word segmentation. Our model is also considered as a way to construct an accurate word n-gram language model directly from characters of arbitrary language, without any "word" indications.
Keywords
Computer Science
Markov Chain Monte Carlo in Practice
7,730 Citations1995Walter R. Gilks, Sylvia Richardson +1 more
Improved backing-off for M-gram language modeling
1,490 Citations2002Reinhard Kneser, Hermann Ney
This paper proposes to use distributions which are especially optimized for the task of back-off, which are quite different from the probability distributions that are usually used for backing-off.
Journal of Child LanguageThe child language data exchange system
872 Citations1985Brian MacWhinney, Catherine E. Snow
The formation of the CHILDES, the governance of the system, the nature of the database, the shape of the coding conventions, and the types of computer programs being developed are detailed.
Artificial IntelligenceHidden semi-Markov models
739 Citations2009Shun‐Zheng Yu
An overview of HSMMs is presented, including modelling, inference, estimation, implementation and applications, which has been applied in thirty scientific and engineering areas, including speech recognition/synthesis, human activity recognition/prediction, handwriting recognition, functional MRI brain mapping, and network anomaly detection.
Journal of the American Statistical AssociationBayesian Methods for Hidden Markov Models
535 Citations2002Steven L. Scott
It is shown how recursive computing allows the statistically efficient use of MCMC output when estimating the hidden states, and the use of log-likelihood for assessing MCMC convergence is illustrated.
A hierarchical Bayesian language model based on Pitman-Yor processes
498 Citations2006Yee Whye Teh
It is shown that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing methods for n-gram language models.
CognitionA Bayesian framework for word segmentation: Exploring the effects of context
439 Citations2009Sharon Goldwater, Thomas L. Griffiths +1 more
The results indicate that taking context into account is important for a statistical word segmentation strategy to be successful, and raise the possibility that even young infants may be able to exploit more subtle statistical patterns than have usually been considered.
The COCOON platform (University of Paris)An estimate of an upper bound for the entropy of English
354 Citations1992Peter F. Brown, Vincent J. Della Pietra +3 more
An estimate of an upper bound of 1.75 bits for the entropy of characters in printed English is presented by constructing a word trigram model and then computing the cross-entropy between this model and a balanced sample of English text.
Contextual dependencies in unsupervised word segmentation
313 Citations2006Sharon Goldwater, Thomas L. Griffiths +1 more
Two new Bayesian word segmentation methods are proposed that assume unigram and bigram models of word dependencies respectively, and the bigram model greatly outperforms the unigrams model (and previous probabilistic models), demonstrating the importance of such dependencies forword segmentation.
Journal of the American Statistical AssociationThe Nested Dirichlet Process
305 Citations2008Abel Rodríguez, David B. Dunson +1 more
Edinburgh Research Explorer (University of Edinburgh)Bayesian Inference for PCFGs via Markov Chain Monte Carlo
280 Citations2007Mark Johnson, Thomas L. Griffiths +1 more
Interpolating between types and tokens by estimating power-law generators
215 Citations2005Sharon Goldwater, Mark Johnson +1 more
It is shown that taking a particular stochastic process - the Pitman-Yor process - as an adaptor justifies the appearance of type frequencies in formal analyses of natural language, and improves the performance of a model for unsupervised learning of morphology.
Improving nonparameteric Bayesian inference
117 Citations2009Mark Johnson, Sharon Goldwater
This paper investigates some of the choices that arise in formulating adaptor grammars and associated inference procedures, and shows that they can have a dramatic impact on performance in an unsupervised word segmentation task.
A Bayesian Interpretation of Interpolated Kneser-Ney
108 Citations2006YW Teh
Open CollectionsParticle Markov chain Monte Carlo
74 Citations2009Roman Holenstein
This thesis proposes a new Monte Carlo framework in which an efficient high-dimensional proposal distributions are built using SMC methods, which allows for effective MCMC algorithms in complex scenarios where standard strategies fail.
Bayesian semi-supervised Chinese word segmentation for statistical machine translation
64 Citations2008Jia Xu, Jianfeng Gao +2 more
A Bayesian semi-supervised Chinese word segmentation model which uses both monolingual and bilingual information to derive a segmentation suitable for MT is proposed and improves a state-of-the-art MT system in a small and a large data environment.
CityU ScholarsAn Empirical Comparison of Goodness Measures for Unsupervised Chinese Word Segmentation with a Unified Framework
52 Citations2008Hai Zhao, Chunyu Kit
Experiments show that description length gain outperforms other measures because of its strength for identifying short words and performance improvement is reported by proper candidate pruning and by assemble segmentation to integrate the strengths of individual measures.
Extended Viterbi algorithm for second order hidden Markov process
36 Citations2003Yang He
An extended Viterbi algorithm is presented that gives a maximum a posteriori estimation of the second-order hidden Markov process that is compared with those of the original first-order one.
Natural Language EngineeringMostly-unsupervised statistical segmentation of Japanese kanji sequences
33 Citations2003Rie Kubota Ando, Lillian Lee
A novel, more robust statistical method utilizing unsegmented training data, which yields performance on long kanji sequences comparable to and sometimes surpassing that of state-of-the-art morphological analyzers over a variety of error metrics.
Semi-Supervised Structured Output Learning Based on a Hybrid Generative and Discriminative Approach
32 Citations2007Jun Suzuki, Akinori Fujino +1 more
This paper proposes a framework for semi-supervised structured output learning (SOL), specifically for sequence labeling, based on a hybrid generative and discriminative approach that significantly outperforms the stateof-the-art performance obtained with supervised SOL methods, such as conditional random fields (CRFs).
The Infinite Markov Model
30 Citations2007Daichi Mochihashi, Eiichiro Sumita
A nonparametric Bayesian method of estimating variable order Markov processes up to a theoretically infinite order by extending a stick-breaking prior to trees of infinite depth associated with a hierarchical Chinese restaurant process, which directly infers the hidden orders of Markov dependencies from which each symbol originated.
Automatic Extraction of New Words from Japanese Texts using Generalized Forward-Backward Search.
11 Citations1996Masaaki Nagata
A novel new word extraction method from Japanese texts based on expected word frequencies is presented using a robust stochastic N-best word segmenter derived from an approximation of the generalized version of the Forward-Backward algorithm.