Decoding Memes: Benchmarking Narrative Role Classification across Multilingual and Multimodal Models

Abstract--This work investigates the challenging task of identifying narrative roles - Hero, Villain, Victim, and Other - in Internet memes, across three diverse test sets spanning English and code-mixed (English-Hindi) languages. Building on an annotated dataset originally skewed toward the'Other' class, we explore a more balanced and linguistically diverse extension, originally introduced as part of the CLEF 2024 shared task. Comprehensive lexical and structural analyses highlight the nuanced, culture-specific, and context-rich language used in real memes, in contrast to synthetically curated hateful content, which exhibits explicit and repetitive lexical markers. T o benchmark the role detection task, we evaluate a wide spectrum of models, including fine-tuned multilingual transformers, sentiment and abuse-aware classifiers, instruction-tuned LLMs, and multimodal vision-language models. Performance is assessed under zero-shot settings using precision, recall, and F1 metrics. W e also explore prompt design strategies to guide multi-modal models and find that hybrid prompts incorporating structured instructions and role definitions offer marginal yet consistent improvements. Our findings underscore the importance of cultural grounding, prompt engineering, and multimodal reasoning in modelling subtle narrative framings in visual-textual content. W arning: This paper contains potentially harmful and offensive content. I. Introduction Social media platforms have become pivotal arenas for rapid information dissemination. However, this openness has also catalysed the proliferation of harmful content - including hate speech, propaganda, and misinformation, often embedded within memes [1], [2]. Memes, with their multimodal structure and cultural resonance, are particularly potent in shaping public opinion and propagating ideologies.