An evolutionary approach to automatic Chinese text segmentation

This proposed method considers Web text a de facto corpus that updates automatically, thus eliminating the need for statistics training and treats the segmentation as a process that finds out the best probability of how individual characters are combined into sentences, paragraphs, and articles.

Abstract

Textual information written in Chinese now represents a huge knowledge repository. The first step of managing and processing information in written Chinese text is segmentation. A new method for automatic Chinese text segmentation using evolutionary algorithms and Web search statistical data is outlined. This proposed method considers Web text a de facto corpus that updates automatically, thus eliminating the need for statistics training. It treats the segmentation as a process that finds out the best probability of how individual characters are combined into sentences, paragraphs, and articles, thus producing segmentation results that are tailored to the text in question and are independent of segmentation standards.