An Improved Chinese Word Segmentation System with Conditional Random Field

Meeting of the Association for Computational LinguisticsPublished 1 July 2006

Hai Zhao, Changning Huang, Mu Li

Citations174

TL;DR

It is found that the use of a 6-tag set, tone feature of Chinese character and assistant segmenters trained on other corpora further improve Chinese word segmentation performance.

Abstract

In this paper, we describe a Chinese word segmentation system that we developed for the Third SIGHAN Chinese Language Processing Bakeoff (Bakeoff2006). We took part in six tracks, namely the closed and open track on three corpora, Academia Sinica (CKIP), City University of Hong Kong (CityU), and University of Pennsylvania/University of Colorado (UPUC). Based on a conditional random field based approach, our word segmenter achieved the highest F measures in four tracks, and the third highest in the other two tracks. We found that the use of a 6-tag set, tone feature of Chinese character and assistant segmenters trained on other corpora further improve Chinese word segmentation performance.

Keywords

Computer Science

ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

12,978 Citations2001John Lafferty, Andrew McCallum +1 more

This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.

View PDF

Chinese segmentation and new word detection using conditional random fields

468 Citations2004Fuchun Peng, Fangfang Feng +1 more

The ability of linear-chain conditional random fields (CRFs) to perform robust and accurate Chinese word segmentation by providing a principled framework that easily supports the integration of domain knowledge in the form of multiple lexicons of characters and words is demonstrated.

View PDF

Chinese Word Segmentation as Character Tagging

376 Citations2003Nianwen Xue

Preliminary results show that this approach is competitive against other supervised machine-learning segmenters reported in previous studies, achieving precision and recall rates of 95.01% and 94.94% respectively, trained on a 237K-word training set.

A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005

337 Citations2005Huihsin Tseng, Pi-Chuan Chang +3 more

A Chinese word segmentation system built using a conditional random field sequence model that provides a framework to use a large number of linguistic features such as character identity, morphological and character reduplication features is presented.

The Second International Chinese Word Segmentation Bakeoff.

308 Citations2005Thomas E. Emerson

The J-splint is a reliable, simple, and rapidly applied splint that prevents many of the complications and downfalls of other described temporizing measures and helps to provide excellent pain management in the acute setting.

View PDF

Computational LinguisticsChinese Word Segmentation and Named Entity Recognition: A Pragmatic Approach

221 Citations2005Jianfeng Gao, Mu Li +2 more

A pragmatic mathematical framework in which segmenting known words and detecting unknown words of different types can be performed simultaneously in a unified way is proposed and implemented in an adaptive Chinese word segmenter, called MSRSeg, which is described in detail.

View PDF

The first international Chinese word segmentation Bakeoff

213 Citations2003Richard Sproat, Thomas E. Emerson

This paper presents the results from the ACL-SIGHAN-sponsored First International Chinese Word Segmentation Bakeoff held in 2003 and reported in conjunction with the Second SIGHAN Workshop on Chinese Language Processing, Sapporo, Japan.

View PDF

A Maximum Entropy Approach to Chinese Word Segmentation

171 Citations2005Jin Kiat Low, Hwee Tou Ng +1 more

This work evaluated the Chinese word segmenter in the open track, on all four corpora, namely Academia Sinica, City University of Hong Kong, Microsoft Research, and Peking University, and achieved the highest F measure for AS, CITYU, and PKU.

Chinese word segmentation as LMR tagging

158 Citations2003Nianwen Xue, Libin Shen

This paper presents Chinese word segmentation algorithms based on the so-called LMR tagging implemented with the Maximum Entropy Markov Model and using Transformation-Based Learning to combine the results of the two LMR taggers that scan the input in opposite directions.

View PDF

Improved source-channel models for Chinese word segmentation

61 Citations2003Jianfeng Gao, Mu Li +1 more

A Chinese word segmentation system that uses improved source-channel models of Chinese sentence generation and provides a unified approach to the four fundamental features of word-level Chinese language processing.

View PDF