Fast Attention Over Long Sequences With Dynamic Sparse Flash Attention

Transformer-based language models have found many diverse applications requiring them to process sequences of increasing length.