Multilingual CLIP with Huggingface + PyTorch Lightning - KDnuggets

This is a walkthrough of training CLIP by OpenAI. CLIP was designed to put both images and text into a new projected space such that they can map to each other by simply looking at dot products. Traditionally training sets like imagenet only allowed you to map images to a single class (and hence one word). This method allows you to map text to images, but can also be used to map images to text if the need arises. This particular blog however is specifically how we managed to train this on colab GPUs using huggingface transformers and pytorch lightning.