Unsupervised segmentation of Chinese text by use of branching entropy

Published 1 January 2006Open access

Zhihui Jin, Kumiko Tanaka‐Ishii

Citations66

TL;DR

An unsupervised segmentation method based on an assumption that the increasing point of entropy of successive characters is the location of a word boundary is proposed, finding that the precision was stable at around 90% independently of the learning data size.

Abstract

We propose an unsupervised segmentation method based on an assumption about language data: that the increasing point of entropy of successive characters is the location of a word boundary. A large-scale experiment was conducted by using 200 MB of unsegmented training data and 1 MB of test data, and precision of 90% was attained with recall being around 80%. Moreover, we found that the precision was stable at around 90% independently of the learning data size.

Keywords

Computer Science

HHMM-based Chinese lexical analyzer ICTCLAS

454 Citations2003Huaping Zhang, Hongkui Yu +2 more

View PDF

The Second International Chinese Word Segmentation Bakeoff.

308 Citations2005Thomas E. Emerson

The J-splint is a reliable, simple, and rapidly applied splint that prevents many of the complications and downfalls of other described temporizing measures and helps to provide excellent pain management in the acute setting.

View PDF

From Phoneme to Morpheme

303 Citations1970Zellig S. Harris

A Maximum Entropy Approach to Chinese Word Segmentation

171 Citations2005Jin Kiat Low, Hwee Tou Ng +1 more

This work evaluated the Chinese word segmenter in the open track, on all four corpora, namely Academia Sinica, City University of Hong Kong, Microsoft Research, and Peking University, and achieved the highest F measure for AS, CITYU, and PKU.

Chinese word segmentation without using lexicon and hand-crafted training data

94 Citations1998Maosong Sun, Shen Dayang +1 more

A new algorithm for segmenting Chinese texts without making use of any lexicon and hand-crafted linguistic resource is presented, that is, mutual information and the difference of t-score between characters is derived automatically from raw Chinese corpora.

Mostly-unsupervised statistical segmentation of Japanese: applications to kanji

36 Citations2000Rie Kubota Ando, Lillian Lee

A novel statistical method utilizing unsegmented training data is introduced, with performance on kanji sequences comparable to and sometimes surpassing that of morphological analyzers over a variety of error metrics.

Proceedings of the 17th international conference on Computational linguistics -Chinese word segmentation without using lexicon and hand-crafted training data

35 Citations1998Maosong Sun, Shen Dayang +1 more

This paper presents a new algoritlml for segmenting Chinese texts without making use of any lexicon and hand-crafted linguistic resource, and the algorithm itself can also be utilized as a stand-alone segmenter in some NLP applications.

View PDF

Lecture notes in computer scienceUnsupervised Segmentation of Chinese Corpus Using Accessor Variety

29 Citations2005Haodi Feng, Kang Chen +2 more

This paper proposes a novel unsupervised method to segment large Chinese corpora using contextual information using the number of characters preceding and following a string to measure the independence of the string.

Lecture notes in computer scienceEntropy as an Indicator of Context Boundaries: An Experiment Using a Web Search Engine

23 Citations2005Kumiko Tanaka‐Ishii

This paper fundamentally verify the extent to which the uncertainty of tokens coming after a sequence helps determine whether a given position is at a context boundary.

Experiments in Unsupervised Entropy-Based Corpus Segmentation

20 Citations1999André Kempe