A Spectral Algorithm for Learning Hidden Markov Models

Hidden Markov Models (HMMs) (Baum and Eagon, 1967; Rabiner, 1989) are the workhorse statistical model for discrete time series, with widely diverse applications including automatic speech recognition, natural language processing (NLP), and genomic sequence modeling. In this model, a discrete hidden state evolves according to some Markovian dynamics, and observations at a particular time depend only on the hidden state at that time. The learning problem is to estimate the model only with observation samples from the underlying distribution. Thus far, the predominant learning algorithms have been local search heuristics, such as the Baum-Welch / EM algorithm (Baum et al., 1970; Dempster et al., 1977). It is not surprising that practical algorithms have resorted to heuristics, as the general learning problem has been shown to be hard under cryptographic assumptions (Terwijn, 2002). Fortunately, the hardness results are for HMMs that seem divorced from those that we are likely to encounter in practical applications. The situation is in many ways analogous to learning mixture distributions with samples from the underlying distribution. There, the general problem is also believed to be hard. However, much recent progress has been made when certain separation assumptions are made with respect to the component mixture distributions (e.g.