Better Generative Models for Sequential Data Problems: Bidirectional Recurrent Mixture Density Networks

This paper describes bidirectional recurrent mixture density networks, which can model multi-modal distributions of the type P(Xt Iyf) and P(Xt lXI, X2,...,Xt-l, yf) without any explicit assumptions about the use of context. These expressions occur frequently in pattern recognition problems with sequential data, for example in speech recognition. Experiments show that the proposed generative models give a higher likelihood on test data compared to a traditional modeling approach, indicating that they can summarize the statistical properties of the data better. 1 Introduction Many problems of engineering interest can be formulated as sequential data problems in an abstract sense as supervised learning from sequential data, where an input vector (dimensionality D) sequence X xf {X!,X2,...,XT_!,XT} living in space X has to be mapped to an output vector (dimensionality J) target sequence T tf {tl' t 2,..., tT