A stochastic finite-state word-segmentation algorithm for Chinese
Published 1 January 1994Open access
Richard Sproat, William A. Gale, Chilin Shih, Nancy Chang
Citations290
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A stochastic finite-state model is presented for segmenting Chinese text into dictionary entries and productively derived words, and providing pronunciations for these words; the method incorporates a class-based model in its treatment of personal names.
Abstract
We present a stochastic finite-state model for segmenting Chinese text into dictionary entries and productively derived words, and providing pronunciations for these words; the method incorporates a class-based model in its treatment of personal names. We also evaluate the system's performance, taking into account the fact that people often do not agree on a single segmentation.
Keywords
Computer Science
Word association norms, mutual information, and lexicography
3,674 Citations1990Kenneth Church, Patrick Hanks
Journal of the American Oriental SocietyMandarin Chinese: A Functional Reference Grammar
2,919 Citations1987Stephen Wadley, Charles N. Li +1 more
Computational LinguisticsRegular models of phonological rule systems
816 Citations1994Ronald M. Kaplan, Martin Kay
This paper shows in detail how this framework applies to ordered sets of context-sensitive rewriting rules and also to grammars in Koskenniemi's two-level formalism.
Helda (University of Helsinki)Two-Level Morphology: A General Computational Model for Word-Form Recognition and Production
792 Citations1983Kimmo Koskenniemi
The New S Language
575 Citations2018Richard A. Becker, John M. Chambers +1 more
This book provides documentation for a new version of the S system released in 1988, which enhances the features that have made S popular: interactive computing, flexible graphics, data management and a large collection of functions.
Computer Speech & LanguageRobust part-of-speech tagging using a hidden Markov model
416 Citations1992Julian Kupiec
A system for part-of-speech tagging is described, based on a hidden Markov model which can be trained using a corpus of untagged text, which results in a model that correctly tags approximately 96% of the text.
The MIT Press eBooksMorphology and Computation
309 Citations1992Richard Sproat
The nature of morphology, the structure of words, and word-formation rules, and the order of morphemes phonology psycholinguistic evidence are discussed.
Computer Speech & LanguageA comparison of the enhanced Good-Turing and deleted estimation methods for estimating probabilities of English bigrams
240 Citations1991Kenneth Church, William A. Gale
The enhanced Good-Turing method is introduced, which is three to four times as efficient in its use of data as the enhanced deleted estimate method and provides accurate predictions of the variances of the standard probabilities estimated from the test corpus.
Artificial IntelligencePitch accent in context predicting intonational prominence from text
231 Citations1993Julia Hirschberg
A series of recent experiments on corpora of recorded (read) speech and spontaneous (elicited) speech suggest that it is indeed possible to model human accent strategies with fair success for unrestricted text—with only the tools for automatic text analysis currently available.
Word identification for Mandarin Chinese sentences
183 Citations1992Keh-Jiann Chen, Shing-Huan Liu
This paper proposes a matching algorithm with 6 different heuristic rules to resolve the ambiguities of Chinese sentences and supports that the maximal matching algorithm is the most effective heuristics.
Two-level morphology with composition
158 Citations1992Lauri Karttunen, Ronald M. Kaplan +1 more
Two-Level Morphology with Composition Lauri Karttunen, Ronald M. Kaplan, and Annie Zaenen Xerox Palo Alto Research Center Center for the Study of language and Information StanJbrd University 1.
Computer Speech & LanguageAutomatic classification of intonational phrase boundaries
149 Citations1992Michelle Q. Wang, Julia Hirschberg
This work investigates the usefulness of a number of textual features and additionalintonational features in predicting the location of one particular intonational feature—intonational phrase boundaries—in natural speech.
Journal of the American Society for Information ScienceChinese text segmentation for text retrieval: Achievements and problems
121 Citations1993Zimin Wu, Gwyneth Tseng
A review of the various different approaches to Chinese text segmentation has been classified in order to provide a general picture of the research activity in this area and demonstrate thatText segmentation remains one of the most challenging and interesting areas for Chinese text retrieval.
A stochastic Japanese morphological analyzer using a forward-DP backward-<i>A*</i> N-best search algorithm
90 Citations1994Masaaki Nagata
A novel method for segmenting the input sentence into words and assigning parts of speech to the words and an efficient two-pass N-best search algorithm is presented, suitable for written Japanese.
Weighted rational transductions and their application to human language processing
72 Citations1994Fernando Pereira, Michael Riley +1 more
The concepts of weighted language, transduction and automaton from algebraic automata theory are presented as a general framework for describing and implementing decoding cascades in speech and language processing and applications to speech recognition and Chinese text segmentation are discussed.
Computer Speech & LanguageEnglish noun-phrase accent prediction for text-to-speech
64 Citations1994Richard Sproat
The various components of np are evaluated, and it is shown that the overall performance of np significantly reduces the error rate in accent assignment over some simple-minded 'baseline' approaches.
Improving Chinese tokenization with linguistic filters on statistical lexical acquisition
62 Citations1994Dekai Wu, Pascale Fung
Empirical evidence is presented for four points concerning tokenization of Chinese text: more rigorous "blind" evaluation methodology is needed to avoid inflated accuracy measurements, the extent of the unknown-word problem is far more serious than generally thought, and the nk-blind method is introduced.
Journal of East Asian LinguisticsA corpus-based analysis of Mandarin nominal root compound
41 Citations1996Richard Sproat, Chilin Shih
A finite-state morphological processor for Spanish
35 Citations1990Evelyne Tzoukermann, Mark Liberman
Recognizing unregistered names for Mandarin word identification
28 Citations1992Liang-Jyh Wang, Wei-Chuan Li +1 more
A new mechanism, based on the concept of sublanguage, is proposed for identifying unknown words, especially personal names, in Chinese newspapers, which includes title-driven name recognition, adaptive dynamic word formation, identification of 2-character and 3-character Chinese names without title.
Constituent-based morphological parsing
5 Citations1987Richard Sproat, Barbara Brunson
A model of morphological processing which directly encodes prosodic constituency, a notion which is clearly crucial in many widespread morphological processes is presented.