This paper discusses the discrete-time Bayesian optimal control of stochastic dynamic systems where some vectors, which augment the system state vectors and the observed state vectors by additional variables, constitute multi-dimensional Markov chains. Optimal control of such Markovian control systems is considered under the assumption that only a part of the components of such vectors is observed by the control system. Certain conditional probability densities needed in deriving optimal control policies are derived, and computational procedures which determine optimal control sequences are given.

MIT Press.