Factored Language Models Tutorial

Published 1 January 2007

Katrin Kirchhoff, Jeff Bilmes, Kevin Duh

Citations32

TL;DR

This tutorial serves as a comprehensive description of FLMs and related algorithms as implemented in the SRI Language Modeling toolkit and provides an introductory walk-through using FLMs on an actual dataset.

Abstract

The Factored Language Model (FLM) is a flexible framework for incorporating various information sources, such as morphology and part-of-speech, into language modeling. FLMs have so far been successfully applied to tasks such as speech recognition and machine translation; it has the potential to be used in a wide variety of problems in estimating probability tables from sparse data. This tutorial serves as a comprehensive description of FLMs and related algorithms. We document the FLM functionalities as implemented in the SRI Language Modeling toolkit and provide an introductory walk-through using FLMs on an actual dataset. Our goal is to provide an easy-to-understand tutorial and reference for researchers interested in applying FLMs to their problems. Overview of the Tutorial We first describe the factored language model (Section 1) and generalized backoff (Section 2), two complementary techniques that attempt to improve statistical estimation (i.e., reduce parameter variance) in language models, and that also attempt to better describe the way in which language (and sequences of words) might be produced. Researchers familar with the algorithms behind FLMs may skip to Section 3, which describes the FLM programs and file formats in the publicly-available SRI Language Modeling (SRILM) toolkit. 1 Section 4 is a step-by-step walkthrough

Keywords

Computer Science

Computational LinguisticsA maximum entropy approach to natural language processing

3,120 Citations1996Adam Berger, Vincent J. Della Pietra +1 more

A maximum-likelihood approach for automatically constructing maximum entropy models is presented and how to implement this approach efficiently is described, using as examples several problems in natural language processing.

Computer Speech & LanguageAn empirical study of smoothing techniques for language modeling

2,077 Citations1999Stanley F. Chen, Joshua Goodman

Journal of the American Statistical AssociationStatistical Methods for Speech Recognition

1,988 Citations1999Don X. Sun, Frederick Jelinek

The speech recognition problem hidden Markov models the acoustic model basic language modelling the Viterbi search hypothesis search on a tree and the fast match elements of information theory.

Factored language models and generalized parallel backoff

283 Citations2003Jeff Bilmes, Katrin Kirchhoff

Initial perplexity results on both CallHome Arabic and on Penn Treebank Wall Street Journal articles are provided, and FLMs with GPB can produce bigrams with significantly lower perplexity, sometimes lower than highly-optimized baseline trigrams.

View PDF

IEEE International Conference on Acoustics Speech and Signal ProcessingThe graphical models toolkit: An open source software system for speech and time-series processing

182 Citations2002Jeff Bilmes, Geoffrey Zweig

The Graphical Models Toolkit is described, an open source, publically available toolkit for developing graphical-model based speech recognition and general time series systems, with features including a language for specifying structures and probability distributions, logarithmic space exact training and decoding procedures, and a generalized EM training method which allows arbitrary sub-Gaussian parameter tying.

The Journal of the Acoustical Society of AmericaGraphical models and automatic speech recognition

88 Citations2002Jeff Bilmes

It is shown that the statistical assumptions behind many pattern recognition techniques commonly used as part of a speech recognition system can be described by a graph — this includes Gaussian distributions, mixture models, decision trees, factor analysis, principle component analysis, linear discriminant analysis, and hidden Markov models.

Automatic learning of language model structure

37 Citations2004Kevin Duh, Katrin Kirchhoff

This paper presents an entirely data-driven model selection procedure based on genetic search, which is shown to outperform both knowledge-based and random selection procedures on two different language modeling tasks (Arabic and Turkish).

View PDF