We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length.

At the core of FiSHformer is a novel finite admixture model of shared heads (FiSH) that samples attention matrices from a set of global attention matrices. The number of global attention matrices is much smaller than the number of local attention matrices generated. FiSHformers directly learn these global attention matrices rather than the local ones as in other transformers, thus significantly improving the computational and memory efficiency of the model.

We introduce the goal of learning a context-agnostic classifier, i.e., a classifier whose predictions are invariant under perturbations of the context.