Chinese Word Segmentation Based on Contextual Entropy

Waseda University Repository (Waseda University)Published 1 October 2003

Jin Huang, David Powers

Citations29

TL;DR

This paper presents a new statistical approach to segment Chinese sequences into words based on contextual entropy on both sides of a bigram to capture the dependency with the left and right contexts in which a bigram occurs.

Abstract

Chinese is written without word delimiters so word segmentation is generally considered a key step in processing Chinese texts. This paper presents a new statistical approach to segment Chinese sequences into words based on contextual entropy on both sides of a bigram. It is used to capture the dependency with the left and right contexts in which a bigram occurs. Our approach tries to segment by finding the word boundaries instead of the words. Experimental results show that it is effective for Chinese word segmentation.

Keywords

Computer Science

From Phoneme to Morpheme

303 Citations1970Zellig S. Harris

A stochastic finite-state word-segmentation algorithm for Chinese

290 Citations1994Richard Sproat, William A. Gale +2 more

A stochastic finite-state model is presented for segmenting Chinese text into dictionary entries and productively derived words, and providing pronunciations for these words; the method incorporates a class-based model in its treatment of personal names.

View PDF

Computational LinguisticsA Compression-based Algorithm for Chinese Word Segmentation

157 Citations2000William J. Teahan, Yingying Wen +2 more

A scheme that infers appropriate positions for word boundaries using an adaptive language model that is standard in text compression, trained on a corpus of presegmented text, and applied to new text so as to maximize the compression obtained is described.

View PDF

A trainable rule-based algorithm for word segmentation

98 Citations1997David D. Palmer

This paper presents a trainable rule-based algorithm for performing word segmentation that provides a simple, language-independent alternative to large-scale lexical-based segmenters requiring large amounts of knowledge engineering.

View PDF

Lecture notes in computer scienceSelf-Supervised Chinese Word Segmentation

57 Citations2001Fuchun Peng, Dale Schuurmans

This procedure uses successive EM phases to learn a good probability model over character strings, and then prunes this model with a mutual information selection criterion to obtain a more accurate word lexicon.

Extraction of Chinese compound words

45 Citations2000Jian Zhang, Jianfeng Gao +1 more

Experimental results show that this method is efficient and robust compared with other approaches, and results on information retrieval to show the usefulness of extracted compounds.

View PDF

USe: A Retargetable Word Segmentation Procedure for Information Retrieval

37 Citations1996Jay Ponte

The experiments show that a fairly simple underlying model can produce reasonable segmentation results, can do so quickly enough to be useful for indexing in an information retrieval system and can be re-targeted to new languages without a great deal of human effort.

A new statistical formula for Chinese text segmentation incorporating contextual information

37 Citations1999Yubin Dai, Teck Ee Loh +1 more

A new statistical formula for identifying 2-character words in Chinese text, called the contextual information formula, was developed empirically by performing stepwise logistic regression using a sample of sentences that had been manually segmented.

Experiments in Unsupervised Entropy-Based Corpus Segmentation

20 Citations1999André Kempe

Automatic Construction of a Chinese Electronic Dictionary.

18 Citations1995Jing-Shin Chang, Yi‐Chung Lin +1 more

A two-class classifier, which is capable of classifying an n-gram either as a word or a non-word, is used in combination with the Viterbi training module to improve the system performance.