Boltzmann Chains and Hidden Markov Models

A statistical mechanical framework for the modeling of discrete time series is proposed, and maximum likelihood estimation is done via Boltzmann learning in one-dimensional networks with tied weights, which motivates new architectures that address particular shortcomings of HMMs.

Abstract

We propose a statistical mechanical framework for the modeling of discrete time series. Maximum likelihood estimation is done via Boltzmann learning in one-dimensional networks with tied weights. We call these networks Boltzmann chains and show that they contain hidden Markov models (HMMs) as a special case. Our framework also motivates new architectures that address particular shortcomings of HMMs. We look at two such architectures: parallel chains that model feature sets with disparate time scales, and looped networks that model long-term dependencies between hidden states. For these networks, we show how to implement the Boltzmann learning rule exactly, in polynomial time, without resort to simulated or mean-field annealing. The necessary computations are done by exact decimation procedures from statistical mechanics. 1 INTRODUCTION AND SUMMARY Statistical models of discrete time series have a wide range of applications, most notably to problems in speech recognition (Juang & Rabin...