Mixed Membership Recurrent Neural Networks

Recurrent neural networks (RNNs) have become one of the standard models in sequential data analysis [Rumelhart et al., 1986, Elman, 1990]. At each time step of the RNN, an observation is modeled via a neural network using the observations and hidden states from previous time points. Models such as the RNN, and also the hidden Markov model among others, often implicitly assume a sequence as having a fixed time interval between observations. They also often do not account for group-level effects when multiple sequences are observed and each sequence belongs to one of multiple groups. For example, consider data in the form of a sequence of discrete counts by a set of groups-- e.g., a sequence of purchases (market baskets) for a set of customers, with one sequence per customer. A vanilla RNN implementation would model these sequences using a network with the same parameters, which removes the customer-level information, and according to an enumerated indexing, which removes the time interval information between orders. However, this information is important: customer-specific effects can improve predictive performance for each customer, while an interval of one day versus one month between orders significantly impacts the items likely to be purchased next.