Remote sensing imagery is dense with objects and contextual visual information. There is a recent trend to combine paired satellite images and text captions for pretraining performant encoders for downstream tasks. However, while contrastive image-text methods like CLIP enable vision-language alignment and zero-shot classification ability, vision-only downstream performance tends to degrade compared to image-only pretraining, such as MAE. In this paper, we propose FLAVARS, a pretraining method that combines the best of both contrastive learning and masked modeling, along with geospatial alignment via contrastive location encoding. We find that FLAVARS significantly outperforms a baseline of SkyCLIP for vision-only tasks such as KNN classification and semantic segmentation, +6\% mIOU on SpaceNet1, while retaining the ability to perform zero-shot classification, unlike MAE pretrained methods.

The image-text contrastive loss resembles that of CLIP. Given a batch of images and text, the cosine similarities between matched image and text pairs are maximized and those for the unmatched pairs are minimized. In this paper, it is found that a noticeable performance gain by performing full backpropagation across GPUs. That's why it is called Global Contrastive (GC) Loss. Given an image and text input, the input image patches are first tokenized using a pretrained dVAE tokenizer, as in DALL·E, which maps each image patch into an index in a visual codebook similar to a word dictionary.