Paper explained: DINO -- Emerging Properties in Self-Supervised Vision Transformers

In this story, I would love to give you a a good idea of how the DINO paper works and what makes it great. I've tried to keep the article simple so that even readers with little prior knowledge can follow along. Traditionally, Vision Transformers (ViT) have not been as attractive as some would expect: They have high computational demands, need more training data, and their features do not exhibit unique properties. With their 2020 paper, "Emerging Properties in Self-Supervised Vision Transformers", Caron et al. aimed to examine why supervised ViT have not yet taken off and if that could be changed by applying self-supervised learning methods to them. This meant that a human would have to create labels for the training data like telling the model that there is a dog in the image.