Estimation and control in Markov chains

Asymptotic properties of the estimate and of the criterion function follow from the law of large numbers and from the central limit theorem for controlled Markov chains derived with the aid of martingales.

Abstract

We consider a finite controlled Markov chain, the description of which depends on an unknown parameter a , and investigate the following control policy. To each a an optimal stationary control is associated. a is estimated recurrently from the trajectory by the minimum contrast method, and the optimal stationary control corresponding to the estimate is used. We present asymptotic properties of the estimate and of the criterion function. They follow from the law of large numbers and from the central limit theorem for controlled Markov chains derived with the aid of martingales.