IBiT: Utilizing Inductive Biases to Create a More Data Efficient Attention Mechanism

In recent years, Transformer-based architectures have become the dominant method for Computer Vision applications. While Transformers are explainable and scale well with dataset size (Dosovitskiy et al., 2020), they lack the inductive biases of Convolutional Neural Networks (LeCun et al., 1989). While these biases may be learned on large datasets, we show that introducing these inductive biases through learned masks allow Vision Transformers to learn on much smaller datasets without Knowledge Distillation. These Transformers, which we call In-ductively Biased Image Transformers (IBiT), are significantly more accurate on small datasets, while retaining the explainability Transformers. Vision Transformers (Dosovitskiy et al., 2020) have become established as the state-of-the-art Computer Vision backbone.