Look Here: Vision Transformers with Directed Attention Generalize and Extrapolate

Our novel method, called LookHere, provides translation-equivariance, ensures attention head diversity, and limits the distribution shift that attention heads face when extrapolating. We demonstrate that LookHere improves performance on classification (avg.