How to get high score using MMBT and CLIP in Hateful Memes Competition

The additional stage of Hateful Memes Competition from Facebook ended a few months ago. My team was lucky enough to take part in this competition and even get pretty good results (we took tenth place). How we did it and what methods we used -- I'll tell you in this article. At first glance, the problem that had to be solved in the competition is quite simple -- to determine whether a meme is hateful or not using text and image data from it. In reality, the problem is complicated by the many ambiguities inherent in our speech, as well as by the presence of sarcasm and irony, with the definition of which neural networks have problems.