Attention can be closely related under certain data conditions to Kanerva's Sparse Distributed Memory (SDM), a biologically plausible associative memory model.

Inthisworkweconstruct interpretable multi-timescale representations by forcing individual units in an LSTMLM to integrate information over specific temporal scales.

Modeling dependencies in multivariate discrete data is a challenging problem, especiallyinhighdimensions.