LLMs for Targeted Sentiment in News Headlines: Exploring the Descriptive-Prescriptive Dilemma
Juroš, Jana, Majer, Laura, Šnajder, Jan
–arXiv.org Artificial Intelligence
News headlines often evoke sentiment by intentionally portraying entities in particular ways, making targeted sentiment analysis (TSA) of headlines a worthwhile but difficult task. Due to its subjectivity, creating TSA datasets can involve various annotation paradigms, from descriptive to prescriptive, either encouraging or limiting subjectivity. LLMs are a good fit for TSA due to their broad linguistic and world knowledge and in-context learning abilities, yet their performance depends on prompt design. In this paper, we compare the accuracy of state-of-the-art LLMs and fine-tuned encoder models for TSA of news headlines using descriptive and prescriptive datasets across several languages. Exploring the descriptive--prescriptive continuum, we analyze how performance is affected by prompt prescriptiveness, ranging from plain zero-shot to elaborate few-shot prompts. Finally, we evaluate the ability of LLMs to quantify uncertainty via calibration error and comparison to human label variation. We find that LLMs outperform fine-tuned encoders on descriptive datasets, while calibration and F1-score generally improve with increased prescriptiveness, yet the optimal level varies.
arXiv.org Artificial Intelligence
May-28-2024
- Country:
- Africa > Middle East
- Asia
- Indonesia (0.04)
- Middle East > Jordan (0.04)
- Europe
- Bulgaria (0.04)
- Croatia
- Dubrovnik-Neretva County > Dubrovnik (0.04)
- Zagreb County > Zagreb (0.04)
- Monaco (0.04)
- North America
- Canada
- British Columbia > Metro Vancouver Regional District
- Vancouver (0.04)
- Ontario > Toronto (0.04)
- British Columbia > Metro Vancouver Regional District
- Dominican Republic (0.04)
- United States > Washington
- King County > Seattle (0.04)
- Canada
- Genre:
- Research Report (0.64)
- Industry:
- Technology: