Convolutional Xformers for Vision

Even though transformers Vaswani et al. [2017], Devlin et al. [2019] have become the state-of-the-art and at par with humans for several natural language processing (NLP) tasks, their applications in vision has been severely limited by their quadratic complexity with respect to sequence length. Even low resolution images, when unrolled, become long 1D sequences of tens of thousands of pixels, and impose a large computational and memory burden on a GPU. A transformer, being a general architecture without an inductive prior, also requires a large number of training images for giving good generalization compared to convolutional models. It also needs extra architectural changes, including the addition of positional embeddings, to gather the positional information of various image pixels. This demand for large amount of data and GPU resources is not suitable for resource-constrained scenarios where data and GPU capabilities are limited, such as green or edge computing Khan et al. [2021]. On the other hand, CNNs have the inductive priors, such as translational equivariance due to convolutional weight sharing and partial scale invariance due to pooling, to handle 2D images which enables them to learn from smaller datasets with less computational expenditure. But, they fail to capture long range dependencies compared to transformers and require deeper networks with several layers to increase their receptive fields. Combining the efficiency and inductive priors of CNNs with the long range information capturing ability of attention can create better architectures that are suitable for computer vision applications.