Context-Aware Pragmatic Metacognitive Prompting for Sarcasm Detection

Abstract--Detecting sarcasm remains a challenging task in the areas of Natural Language Processing (NLP) despite recent advances in neural network approaches. Currently, Pre-trained Language Models (PLMs) and Large Language Models (LLMs) are the preferred approach for sarcasm detection. However, the complexity of sarcastic text, combined with linguistic diversity and cultural variation across communities, has made the task more difficult even for PLMs and LLMs. Beyond that, those models also exhibit unreliable detection of words or tokens that require extra grounding for analysis. Building on a state-of-the-art prompting method in LLMs for sarcasm detection called Pragmatic Metacognitive Prompting (PMP), we introduce a retrieval-aware approach that incorporates retrieved contextual information for each target text. Our pipeline explores two complementary ways to provide context: adding non-parametric knowledge using web-based retrieval when the model lacks necessary background, and eliciting the model's own internal knowledge for a self-knowledge awareness strategy. We evaluated our approach with three datasets, such as Twitter Indonesia Sarcastic, SemEval-2018 T ask 3, and MUStARD. Non-parametric retrieval resulted in a significant 9.87% macro-F1 improvement on Twitter Indonesia Sarcastic compared to the original PMP method. Self-knowledge retrieval improves macro-F1 by 3.29% on Semeval and by 4.08% on MUStARD. Future work will focus on optimizing the retrieval of relevant contextual information and examining how retrieval quality affects performance. In the field of machine learning, natural language processing (NLP) tasks have been shown to be a crucial part of human life. NLP tasks revolve around processing text in a specific manner and receiving an output that can be useful, with examples such as text classification, text generation, information retrieval, and similar related tasks. Sarcasm detection, or as some call it verbal irony detection, is a task in NLP that automatically classifies text, and in extended forms includes images, audio, or video, as either sarcastic or not. Systems designed for sarcasm detection are becoming more important in the 21st century due to the growth of the use of sarcasm detection datasets caused by media usage of it, such as social media, television, and much more. Additionally, enhancing these automatic systems for sarcasm detection could become crucial in interpreting the real sentence meaning of a text.