Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters

Others have invented alternative sequence mixing architectures such as state-space models which are designed The self-attention operation is the core computational to be efficiently computable in linear time and constant building block of the transformer architecture [1, 2], memory [25-29]. Other approaches utilize efficient algorithms which has become an ubiquitous and highly effective to reduce the computational burden of attention workhorse architecture currently applied at scale to language while keeping the core computation the same. These include [3-7], vision [8], audio [9], and decision-making memory-efficient attention [30], Flash Attention [10, 11]. Nonetheless, the quadratic time complexity of [31] and Flash Decoding [32], which provide a set of IOaware self-attention means that significant resources are required kernels to map the attention operation to the GPU to train and generate from transformer-based Large Language hardware resources in an extremely efficient way, significantly Models (LLMs), especially for models with large reducing the memory overhead required.