Lexicalized hidden Markov models for part-of-speech tagging

Published 1 January 2000Open access

Sang-Zoo Lee, Jun’ichi Tsujii, Hae‐Chang Rim

Citations31

TL;DR

This paper introduces uniformly lexicalized HMMs for part-of-speech tagging in both English and Korean and uses a simplified back-off smoothing technique to overcome data sparseness.

Abstract

Since most previous works for HMM-based tagging consider only part-of-speech information in contexts, their models cannot utilize lexical information which is crucial for resolving some morphological ambiguity. In this paper we introduce uniformly lexicalized HMMs for part-of-speech tagging in both English and Korean. The lexicalized models use a simplified back-off smoothing technique to overcome data sparseness. In experiments, lexicalized models achieve higher accuracy than non-lexicalized models and the back-off smoothing method mitigates data sparseness better than simple smoothing methods.

Keywords

Computer Science

Pattern classification and scene analysis

12,643 Citations1973Richard O. Duda, Peter E. Hart

Proceedings of the IEEEThe viterbi algorithm

5,595 Citations1973G. David Forney

This paper gives a tutorial exposition of the Viterbi algorithm and of how it is implemented and analyzed, and increasing use of the algorithm in a widening variety of areas is foreseen.

BiometrikaTHE POPULATION FREQUENCIES OF SPECIES AND THE ESTIMATION OF POPULATION PARAMETERS

3,227 Citations1953I. J. Good

IEEE Transactions on Acoustics Speech and Signal ProcessingEstimation of probabilities from sparse data for the language model component of a speech recognizer

1,645 Citations1987Slava M. Katz

The model offers, via a nonlinear recursive procedure, a computation and space efficient solution to the problem of estimating probabilities from sparse data, and compares favorably to other proposed methods.

Maximum Entropy Model for Part-Of-Speech Tagging

1,276 Citations1996Adwait Ratnaparkhi

A statistical model which trains from a corpus annotated with Part Of Speech tags and assigns them to previously unseen text with state of the art accuracy and discusses the corpus consistency problems discovered during the implementation of these features.

arXiv (Cornell University)Some Advances in Transformation-Based Part of Speech Tagging

279 Citations1994Eric Brill

View PDF

Equations for part-of-speech tagging

169 Citations1993Eugene Charniak, Curtis Hendrickson +2 more

This paper derives from first principles the basic equations for a few of the basic hidden-Markov-model word taggers as well as equations for other models which may be novel (the descriptions in previous papers being too spare to be sure).

arXiv (Cornell University)Building Probabilistic Models for Natural Language

93 Citations1996Stanley F. Chen

This thesis investigates three problems involving the probabilistic modeling of language: smoothing n-gram models, statistical grammar induction, and bilingual sentence alignment, and relates each of these frameworks to the Bayesian paradigm, and shows why each framework was appropriate for the given problem.

View PDF

Tagging text with a probabilistic model

79 Citations1991Bernard Mérialdo

Experiments show that the best training is obtained by using as much tagged text as is available, and a maximum likelihood training may improve the accuracy of the tagging.

arXiv (Cornell University)HMM Specialization with Selective Lexicalization

9 Citations1999Jin-Dong Kim, Sang-Zoo Lee +1 more

A technique which complements Hidden Markov Models by incorporating some lexicalized states representing syntactically uncommon words is presented, which improved the tagging accuracy by 0.21% at the 95% level of confidence.

View PDF