Variational Recurrent Neural Networks -- VRNNs

First of all, Why VRNN? -- It's the result of the attempt to include the latent random variables into the hidden state of the RNN by combining the elements of the variational autoencoder. Learning generative models for sequences is a very challenging task. Significant work in this direction exists because of Dynamic Bayesian Networks (DBNs) such as Hidden Markov Models (HMMs) and Kalman Filters, but the dominance of DBN-based approaches has now been recently overturned by an interest in the recurrent neural network-based approaches. We know that RNN is very special in the sense that it is able to handle both the variable-length input and output and, by training an RNN to predict the next output in a sequence, given all the previous outputs, it can be used to model joint probability distribution over sequences. RNNs possess both a richly distributed internal state representation and flexible non-linear transition functions (which determine the evolution of the internal hidden state) giving them high expressive power and as a consequence of which RNNs have gained significant popularity as generative models for highly structured sequential data such as natural speech. By highly structured data, the authors meant that the data is characterized by two properties.