Pretraining the Vision Transformer using self-supervised methods for vision based Deep Reinforcement Learning

In recent years, a new architecture for vision-based tasks that does not use convolutions called the Vision Transformer (ViT) (Dosovitskiy et al., 2020) has shown impressive results in several benchmarks. This architecture presents much weaker inductive biases when compared to a CNN, which can result in lower data efficiency. The Vision Transformer, unlike the CNNs, can capture relations between parts of an image (patches) that are far apart from each other, thus deriving global information that can help the model perform better in certain tasks. When the model is pretrained, using supervised or self-supervised learning, it manages to surpass in some cases the best convolution-based models in terms of task performance. Nonetheless, despite the successes obtained in computer vision these results are yet to be seen in reinforcement learning. Moreover, while some areas of machine learning have transitioned to large pretrained models, current Deep RL research is still largely based on small neural networks that are trained from tabula rasa. Despite the successes of deep reinforcement learning agents in the last decade, these still require a large amount of data or interactions to learn good policies. This data inefficiency makes current methods difficult to apply to environments where interactions are more expensive or data is scarce, which is the case in many real-world applications. In environments where the agent does not have full access to the current state, i.e. partially observable environments, this problem becomes even more prominent, since the agent not only needs to learn the state-to-action mapping but also a state representation function that tries to be informative about the current state given an observation.