Unsupervized Word Segmentation: the case for Mandarin Chinese
Published 7 December 2013
Pierre Magistry, Benoît Sagot
Citations24
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An unsupervized segmentation system tested on Mandarin Chinese is presented, improving on (Jin and Tanaka-Ishii, 2006) by adding normalization and viterbi-decoding and reaching near state-of-the-art results (Wang et al., 2011) with a simpler system.
Abstract
In this paper, we present an unsupervized segmentation system tested on Mandarin Chinese. Following Harris's Hypothesis in Kempe (1999) and Tanaka-Ishii's (2005) reformulation, we base our work on the Variation of Branching Entropy. We improve on (Jin and Tanaka-Ishii, 2006) by adding normalization and viterbidecoding. This enable us to remove most of the thresholds and parameters from their model and to reach near state-of-the-art results (Wang et al., 2011) with a simpler system. We provide evaluation on different corpora available from the Segmentation bake-off II (Emerson, 2005) and define a more precise topline for the task using cross-trained supervized system available off-the-shelf (Zhang and Clark, 2010; Zhao and Kit, 2008; Huang and Zhao, 2007)
Keywords
Computer Science
Contextual dependencies in unsupervised word segmentation
313 Citations2006Sharon Goldwater, Thomas L. Griffiths +1 more
Two new Bayesian word segmentation methods are proposed that assume unigram and bigram models of word dependencies respectively, and the bigram model greatly outperforms the unigrams model (and previous probabilistic models), demonstrating the importance of such dependencies forword segmentation.
The Second International Chinese Word Segmentation Bakeoff.
308 Citations2005Thomas E. Emerson
The J-splint is a reliable, simple, and rapidly applied splint that prevents many of the complications and downfalls of other described temporizing measures and helps to provide excellent pain management in the acute setting.
From Phoneme to Morpheme
303 Citations1970Zellig S. Harris
Bayesian unsupervised word segmentation with nested Pitman-Yor language modeling
242 Citations2009Daichi Mochihashi, Takeshi Yamada +1 more
A new Bayesian model for fully unsupervised word segmentation and an efficient blocked Gibbs sampler combined with dynamic programming for inference is proposed and confirmed that it significantly outperforms previous reported results in both phonetic transcripts and standard datasets for Chinese and Japaneseword segmentation.
Zhongwen xinxi xuebaoChinese Word Segmentation: A Decade Review
141 Citations2007Hai Zhao
During the last decade, especially since the First International Chinese Word Segmentation Bakeoff was held in July 2003, the study in automatic Chinese word segmentation has been greatly improved.
Computational LinguisticsAccessor Variety Criteria for Chinese Word Extraction
125 Citations2004Haodi Feng, Kang Chen +2 more
This work defines a word to be a meaningful string composed of several Chinese characters, considers the number of distinct predecessors and successors of a string in a large corpus (TREC 5 and TREC 6 documents), and uses them as the measurement of the context independency of astring from the rest of the sentences in the document.
A Fast Decoder for Joint Word Segmentation and POS-Tagging Using a Single Discriminative Model
86 Citations2010Yue Zhang, Stephen Clark
It is shown that the standard beam-search algorithm can be used as an efficient decoder for the global linear model of Zhang and Clark (2008) for joint word segmentation and POS-tagging, achieving a significant speed improvement.
Unsupervised segmentation of Chinese text by use of branching entropy
66 Citations2006Zhihui Jin, Kumiko Tanaka‐Ishii
An unsupervised segmentation method based on an assumption that the increasing point of entropy of successive characters is the location of a word boundary is proposed, finding that the precision was stable at around 90% independently of the learning data size.
CityU ScholarsAn Empirical Comparison of Goodness Measures for Unsupervised Chinese Word Segmentation with a Unified Framework
52 Citations2008Hai Zhao, Chunyu Kit
Experiments show that description length gain outperforms other measures because of its strength for identifying short words and performance improvement is reported by proper candidate pruning and by assemble segmentation to integrate the strengths of individual measures.
Computational LinguisticsA New Unsupervised Approach to Word Segmentation
27 Citations2011Hanshi Wang, Jian Zhu +2 more
The results suggest that ESA is effective on Chinese corpora and the empirical formulae based on the results can be used to predict the parameter in ESA to avoid parameter estimation that is usually time-consuming.
Lecture notes in computer scienceEntropy as an Indicator of Context Boundaries: An Experiment Using a Web Search Engine
23 Citations2005Kumiko Tanaka‐Ishii
This paper fundamentally verify the extent to which the uncertainty of tokens coming after a sequence helps determine whether a given position is at a context boundary.
Experiments in Unsupervised Entropy-Based Corpus Segmentation
20 Citations1999André Kempe
Lecture notes in computer scienceAn Unsupervised Algorithm for Segmenting Categorical Timeseries into Episodes
19 Citations2002Paul R. Cohen, Brent Heeringa +1 more
The Voting-Experts algorithm first collects statistics about the frequency and boundary entropy of ngrams, then passes a window over the series and has two "expert methods" decide where in the window boundaries should be drawn.
HAL (Le Centre pour la Communication Scientifique Directe)Segmentation et induction de lexique non-supervisées du mandarin
2 Citations2011Pierre Magistry, Benoît Sagot
Qu’un modèle simple qui repose sur une reformulation en termes d’entropie d‘une hypothèse indépendante de la langue énoncée par Harris (1955), permet of segmenter un corpus and d”en extraire un lexique.