Lexicon Optimization for Chinese Language Modeling
Published 1 October 2000
Jun Zhao, Jianfeng Gao, Eric Chang, Mingjing Li
Citations6
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The method is an iterative procedure consisting of two phases, namely lexicon generation and lexicon pruning, which prune the lexicon to a preset memory limitation using a perplexity minimization criterion.
Abstract
In this paper, we present an approach to lexicon optimization for Chinese language modeling. The method is an iterative procedure consisting of two phases, namely lexicon generation and lexicon pruning. In the first phase, we extract appropriate new words from a very large training corpus by statistical approaches. In the second phase, we prune the lexicon to a preset memory limitation using a perplexity minimization criterion. Experimental results show up to 6% character perplexity reduction comparing to the baseline lexicon.
Keywords
Computer Science
Statistical language modeling using the CMU-cambridge toolkit
564 Citations1997Philip Clarkson, Roni Rosenfeld
The conventional language modeling technology, as implemented in the toolkit, is outlined, and the extra e(cid:14)ciency and functionality that the new toolkit provides as compared to previous software for this task is described.
A hidden Markov model information retrieval system
469 Citations1999David R. Miller, Tim Leek +1 more
A novel method for performing blind feedback in the HMM framework, a more complex HMM that models bigram production, and several other algorithmic re nements form a state-of-the-art retrieval system that ranked among the best on the TREC-7 ad hoc retrieval task.
arXiv (Cornell University)Entropy-based Pruning of Backoff Language Models
267 Citations2000Andreas Stolcke
It is shown that the relative entropy resulting from pruning a single N-gram can be computed exactly and efficiently for backoff models and shown that a production-quality Hub4 LM can be reduced to 26% its original size without increasing recognition error.
Information Processing & ManagementPAT-tree-based adaptive keyphrase extraction for intelligent Chinese information retrieval
77 Citations1999Lee‐Feng Chien
The proposed PAT-tree-based adaptive approach can reduce the reliance on rigid lexicon and sophisticated word segmentation, and compared with conventional statistics-based approaches, it can handle phrases composed of high-frequency words regardless of phrase length.
Machine TranslationLarge-scale automatic extraction of an English-Chinese translation lexicon
63 Citations1995Dekai Wu, Xuanyin Xia
This work introduces a fully automatic two-stage statistical methodology that is able to learn translations for collocations and reports experimental results on automatic extraction of an English-Chinese translation lexicon, by statistical analysis of a large parallel corpus.
Extraction of Chinese compound words
45 Citations2000Jian Zhang, Jianfeng Gao +1 more
Experimental results show that this method is efficient and robust compared with other approaches, and results on information retrieval to show the usefulness of extracted compounds.
Compound words in large-vocabulary German speech recognition systems
43 Citations2002André Berton, Pablo Fetter +1 more
A new method for handling the components of compounds rather than the compounds themselves is introduced, which not only reduces the vocabulary, and therefore the complexity, but also improves word accuracy, and reduced complexity means a more robust language model.
Phrase bigrams for continuous speech recognition
38 Citations2002E. Giachin
Two procedures for automatically determining frequent phrases (within the framework of a probabilistic language model) in an unlabeled training set of written sentences are discussed and one procedure is optimal since it minimises the set perplexity.
IEEE ExpertNavigating the Information Superhighway using spoken language interfaces
28 Citations1995Victor W. Zue
MIT's Galaxy system provides an intuitive, speech-based user interface that enables universal information access using spoken dialogue that differs from other spoken language approaches because it is distributed and decentralized, multidomain, and extensive.
A unified approach to statistical language modeling for Chinese
23 Citations2002Jianfeng Gao, Haifeng Wang +2 more
The paper presents a unified approach to Chinese statistical language modeling, which automatically and consistently gathers a high-quality training data set from the Web, creates ahigh-quality lexicon, and segments the training data using this lexicon all using a maximum likelihood principle, which is consistent with the trigram training.
Statistics-based segment pattern lexicon-a new direction for Chinese language modeling
16 Citations2002Kae-Cherng Yang, Tai-Hsuan Ho +2 more
This paper presents a new direction for Chinese language modeling based on a different concept of the lexicon, which can be words or any other "segment patterns" that are extracted from the training corpus by statistical approaches with a goal to minimize the overall perplexity.