Using statistical and contextual information to identify two‐ and three‐character words in Chinese text
Journal of the American Society for Information Science and TechnologyPublished 1 January 2002
Christopher S. G. Khoo, Yubin Dai, Teck Ee Loh
Citations20
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
contextual information formula for 2 character words provides significantly better results than previous formulations and using both the 2 and 3 character formulations in combination in combination significantly improves the 2 character results.
Abstract
Abstract New statistical formulas were developed for identifying two‐ and three‐character words in Chinese text. The formulas were constructed by performing stepwise logistic regression using a sample of sentences that had been manually segmented. For identifying two‐character words, the relative frequency of the adjacent characters and the document frequency of the overlapping bigrams were found to be significant factors. These provide information about the immediate neighborhood or context of the character string. Contextual information was also found to be significant in predicting three‐character words. Local information (the number of times the bigram or trigram occurs in the document being segmented) and the position of the bigram/trigram in the sentence were not found to be useful in identifying words. The new formulas, called contextual information formulas , were found to be substantially better than the mutual information formula in identifying two‐ and three‐character words. Using the contextual information formulas for both two‐ and three‐character words gave significantly better results than using the formula for two‐character words alone. The method can also be used for identifying multiword terms in English text.
Keywords
Computer Science
The MIT Press eBooksNatural Language Understanding
1,279 Citations2021
The text features a new chapter on statistically-based methods using large corpora and an appendix on speech recognition and spoken language understanding and information on semantics that was covered in the first edition has been largely expanded in this edition.
Word association norms, mutual information, and lexicography
1,066 Citations1989Kenneth Church, Patrick Hanks
The proposed measure, the association ratio, estimates word association norms directly from computer readable corpora, making it possible to estimate norms for tens of thousands of words.
Querying across languages
291 Citations1996David A. Hull, Gregory Grefenstette
Using translated queries and a bilingual transfer dictionary, it is learned that crosslartguage multilingual IR is feasible, although performance lags considerably behind the monolingual standard.
A stochastic finite-state word-segmentation algorithm for Chinese
290 Citations1994Richard Sproat, William A. Gale +2 more
A stochastic finite-state model is presented for segmenting Chinese text into dictionary entries and productively derived words, and providing pronunciations for these words; the method incorporates a class-based model in its treatment of personal names.
Lecture notes in computer scienceThe C-value/NC-value Method of Automatic Recognition for Multi-word Terms
201 Citations1998Katerina T. Frantzi, Sophia Ananiadou +1 more
This paper presents a domain-independent method for the automatic extraction of multi-word terms, from machine-readable special language corpora, using C-value/NC-value, which enhances the common statistical measure of frequency of occurrence for term extraction, making it sensitive to a particular type ofMulti- word terms, the nested terms.
Computational LinguisticsA Compression-based Algorithm for Chinese Word Segmentation
157 Citations2000William J. Teahan, Yingying Wen +2 more
A scheme that infers appropriate positions for word boundaries using an adaptive language model that is standard in text compression, trained on a corpus of presegmented text, and applied to new text so as to maximize the compression obtained is described.
ACM Transactions on Asian Language Information ProcessingToward a unified approach to statistical language modeling for Chinese
147 Citations2002Jianfeng Gao, Joshua Goodman +2 more
This article presents a unified approach to Chinese statistical language modeling, which automatically and consistently gathers a high-quality training data set from the Web, creates ahigh-quality lexicon, segments the training data using this Lexicon, and compresses the language model by using the maximum likelihood principle, which is consistent with trigram model training.
Information Processing & ManagementGenerating and evaluating domain-oriented multi-word terms from texts
136 Citations1993Fred J. Damerau
This paper uses a surrogate for experimentation by relying on pre-existing dictionaries as indicators of domain relevance and an attempt is made to provide a more objective evaluation of the selection procedures used.
Journal of the American Society for Information ScienceChinese text segmentation for text retrieval: Achievements and problems
121 Citations1993Zimin Wu, Gwyneth Tseng
A review of the various different approaches to Chinese text segmentation has been classified in order to provide a general picture of the research activity in this area and demonstrate thatText segmentation remains one of the most challenging and interesting areas for Chinese text retrieval.
Journal of the American Society for Information ScienceACTS: An automatic Chinese text segmentation system for full text retrieval
53 Citations1995Zimin Wu, Gwyneth Tseng
ACTS is an automatic Chinese text segmentation proto-type for Chinese full text retrieval that applies partial syntactic analysis—the analysis of morphemes, words, and phrases.
Journal of the American Society for Information ScienceCombination and boundary detection approaches on Chinese indexing
39 Citations2000Christopher C. Yang, Johnny Luk +2 more
This article investigates the combination and boundary detection approaches based on mutual information for segmentation in Chinese indexing, which is the foundation of Chinese and cross-lingual information retrieval.
Journal of the American Society for Information ScienceCross-language information access to multilingual collections on the internet
39 Citations2000Guo-Wei Bian, Hsin‐Hsi Chen
This paper deals with query translation and document translation in a Chinese‐English information retrieval system called MTIR, where bilingual dictionary and monolingual corpus‐based approaches are adopted to select suitable translated query terms and a machine transliteration algorithm is introduced to resolve proper name searching.
Discovering Chinese words from unsegmented text (poster abstract)
37 Citations1999Xianping Ge, Wanda Pratt +1 more
This paper investigates an efficient algorithm to discover the words and their occurrence probabilities from a corpus of unsegmented text without using a dictionary using the maximum likelihood principle.
Overlapping statistical word indexing
32 Citations1997Yasushi Ogawa, Toru Matsuda
This paper proposes a segmentation method for Japanese text which uses statistical information of characters which needs only a small amount of statistic information and computation, and does not need constant maintenance.
Journal of the American Society for Information ScienceCross-language information access to multilingual collections on the internet
25 Citations2000Guo-Wei Bian, Hsin‐Hsi Chen
A unified approach to statistical language modeling for Chinese
23 Citations2002Jianfeng Gao, Haifeng Wang +2 more
The paper presents a unified approach to Chinese statistical language modeling, which automatically and consistently gathers a high-quality training data set from the Web, creates ahigh-quality lexicon, and segments the training data using this lexicon all using a maximum likelihood principle, which is consistent with the trigram training.
Journal of the American Society for Information ScienceText segmentation for Chinese spell checking
8 Citations1999Kin Hong Lee, Mau Kit Michael Ng +1 more
A Block-of-Combinations (BOC) segmentation method based on frequency of word usage is proposed to reduce the word combinations from exponential growth to linear growth and to make the segmentation more suitable for spell checking, user interaction is suggested.
Journal of the American Society for Information ScienceText segmentation for Chinese spell checking
6 Citations1999Kin Hong Lee, Mau Kit Michael Ng +1 more
Lecture notes in computer scienceAn Efficient Chinese Word Segmentation Algorithm for Chinese Information Processing on the Internet
1 Citations1999Peter Kok-Yiu Wong
The proposed Chinese word segmentation algorithm based on forward maximum matching and word binding force can be applied to process the huge amount of Chinese information on the Internet.