Government
Motamot: A Dataset for Revealing the Supremacy of Large Language Models over Transformer Models in Bengali Political Sentiment Analysis
Faria, Fatema Tuj Johora, Moin, Mukaffi Bin, Mumu, Rabeya Islam, Abir, Md Mahabubul Alam, Alfy, Abrar Nawar, Alam, Mohammad Shafiul
Sentiment analysis is the process of identifying and categorizing people's emotions or opinions regarding various topics. Analyzing political sentiment is critical for understanding the complexities of public opinion processes, especially during election seasons. It gives significant information on voter preferences, attitudes, and current trends. In this study, we investigate political sentiment analysis during Bangladeshi elections, specifically examining how effectively Pre-trained Language Models (PLMs) and Large Language Models (LLMs) capture complex sentiment characteristics. Our study centers on the creation of the "Motamot" dataset, comprising 7,058 instances annotated with positive and negative sentiments, sourced from diverse online newspaper portals, forming a comprehensive resource for political sentiment analysis. We meticulously evaluate the performance of various PLMs including BanglaBERT, Bangla BERT Base, XLM-RoBERTa, mBERT, and sahajBERT, alongside LLMs such as Gemini 1.5 Pro and GPT 3.5 Turbo. Moreover, we explore zero-shot and few-shot learning strategies to enhance our understanding of political sentiment analysis methodologies. Our findings underscore BanglaBERT's commendable accuracy of 88.10% among PLMs. However, the exploration into LLMs reveals even more promising results. Through the adept application of Few-Shot learning techniques, Gemini 1.5 Pro achieves an impressive accuracy of 96.33%, surpassing the remarkable performance of GPT 3.5 Turbo, which stands at 94%. This underscores Gemini 1.5 Pro's status as the superior performer in this comparison.
Overview of PerpectiveArg2024: The First Shared Task on Perspective Argument Retrieval
Falk, Neele, Waldis, Andreas, Gurevych, Iryna
Argument retrieval is the task of finding relevant arguments for a given query. While existing approaches rely solely on the semantic alignment of queries and arguments, this first shared task on perspective argument retrieval incorporates perspectives during retrieval, accounting for latent influences in argumentation. We present a novel multilingual dataset covering demographic and socio-cultural (socio) variables, such as age, gender, and political attitude, representing minority and majority groups in society. We distinguish between three scenarios to explore how retrieval systems consider explicitly (in both query and corpus) and implicitly (only in query) formulated perspectives. This paper provides an overview of this shared task and summarizes the results of the six submitted systems. We find substantial challenges in incorporating perspectivism, especially when aiming for personalization based solely on the text of arguments without explicitly providing socio profiles. Moreover, retrieval systems tend to be biased towards the majority group but partially mitigate bias for the female gender. While we bootstrap perspective argument retrieval, further research is essential to optimize retrieval systems to facilitate personalization and reduce polarization.
Are LLMs Good Annotators for Discourse-level Event Relation Extraction?
Wei, Kangda, Gautam, Aayush, Huang, Ruihong
Large Language Models (LLMs) have demonstrated proficiency in a wide array of natural language processing tasks. However, its effectiveness over discourse-level event relation extraction (ERE) tasks remains unexplored. In this paper, we assess the effectiveness of LLMs in addressing discourse-level ERE tasks characterized by lengthy documents and intricate relations encompassing coreference, temporal, causal, and subevent types. Evaluation is conducted using an commercial model, GPT-3.5, and an open-source model, LLaMA-2. Our study reveals a notable underperformance of LLMs compared to the baseline established through supervised learning. Although Supervised Fine-Tuning (SFT) can improve LLMs performance, it does not scale well compared to the smaller supervised baseline model. Our quantitative and qualitative analysis shows that LLMs have several weaknesses when applied for extracting event relations, including a tendency to fabricate event mentions, and failures to capture transitivity rules among relations, detect long distance relations, or comprehend contexts with dense event mentions.
Foundations for Unfairness in Anomaly Detection -- Case Studies in Facial Imaging Data
Livanos, Michael, Davidson, Ian
Deep anomaly detection (AD) is perhaps the most controversial of data analytic tasks as it identifies entities that are then specifically targeted for further investigation or exclusion. Also controversial is the application of AI to facial imaging data. This work explores the intersection of these two areas to understand two core questions: "Who" these algorithms are being unfair to and equally important "Why". Recent work has shown that deep AD can be unfair to different groups despite being unsupervised with a recent study showing that for portraits of people: men of color are far more likely to be chosen to be outliers. We study the two main categories of AD algorithms: autoencoder-based and single-class-based which effectively try to compress all the instances with those that can not be easily compressed being deemed to be outliers. We experimentally verify sources of unfairness such as the under-representation of a group (e.g. people of color are relatively rare), spurious group features (e.g. men are often photographed with hats), and group labeling noise (e.g. race is subjective). We conjecture that lack of compressibility is the main foundation and the others cause it but experimental results show otherwise and we present a natural hierarchy amongst them.
IDEA: A Flexible Framework of Certified Unlearning for Graph Neural Networks
Dong, Yushun, Zhang, Binchi, Lei, Zhenyu, Zou, Na, Li, Jundong
Graph Neural Networks (GNNs) have been increasingly deployed in a plethora of applications. However, the graph data used for training may contain sensitive personal information of the involved individuals. Once trained, GNNs typically encode such information in their learnable parameters. As a consequence, privacy leakage may happen when the trained GNNs are deployed and exposed to potential attackers. Facing such a threat, machine unlearning for GNNs has become an emerging technique that aims to remove certain personal information from a trained GNN. Among these techniques, certified unlearning stands out, as it provides a solid theoretical guarantee of the information removal effectiveness. Nevertheless, most of the existing certified unlearning methods for GNNs are only designed to handle node and edge unlearning requests. Meanwhile, these approaches are usually tailored for either a specific design of GNN or a specially designed training objective. These disadvantages significantly jeopardize their flexibility. In this paper, we propose a principled framework named IDEA to achieve flexible and certified unlearning for GNNs. Specifically, we first instantiate four types of unlearning requests on graphs, and then we propose an approximation approach to flexibly handle these unlearning requests over diverse GNNs. We further provide theoretical guarantee of the effectiveness for the proposed approach as a certification. Different from existing alternatives, IDEA is not designed for any specific GNNs or optimization objectives to perform certified unlearning, and thus can be easily generalized. Extensive experiments on real-world datasets demonstrate the superiority of IDEA in multiple key perspectives.
Mastering 'the art of brainwashing,' China intensifies AI censorship
China has once again extended its policy of censorship and surveillance as it looks to keep artificial intelligence (AI) models in check even as it races to advance the ever-expanding technology. The Chinese Communist Party (CCP) has introduced more regulative measures to make sure its home-based tech companies adhere to the party's ideological rules. All AI firms are required to participate in a government review which analyzes the companies' large language models (LLMs) to ensure they "embody core socialist values," as first reported by the Financial Times last week. A man walks past a photo of Chinese President Xi Jinping at the Museum of the Communist Party of China in Beijing on March 3, 2023. A NEW BREED OF MILITARY AI ROBO-DOGS COULD BE MARINES' NEW SECRET WEAPON China has long worked to suppress information accessible over the internet through the use of its "Great Firewall" -- which has been used to block a litany of items perceived as bad for the CCP, such as information surrounding the 1989 Tiananmen Square massacre or memes comparing Chinese President Xi Jinping to Winnie the Pooh.
EaTVul: ChatGPT-based Evasion Attack Against Software Vulnerability Detection
Liu, Shigang, Cao, Di, Kim, Junae, Abraham, Tamas, Montague, Paul, Camtepe, Seyit, Zhang, Jun, Xiang, Yang
Recently, deep learning has demonstrated promising results in enhancing the accuracy of vulnerability detection and identifying vulnerabilities in software. However, these techniques are still vulnerable to attacks. Adversarial examples can exploit vulnerabilities within deep neural networks, posing a significant threat to system security. This study showcases the susceptibility of deep learning models to adversarial attacks, which can achieve 100% attack success rate (refer to Table 5). The proposed method, EaTVul, encompasses six stages: identification of important samples using support vector machines, identification of important features using the attention mechanism, generation of adversarial data based on these features using ChatGPT, preparation of an adversarial attack pool, selection of seed data using a fuzzy genetic algorithm, and the execution of an evasion attack. Extensive experiments demonstrate the effectiveness of EaTVul, achieving an attack success rate of more than 83% when the snippet size is greater than 2. Furthermore, in most cases with a snippet size of 4, EaTVul achieves a 100% attack success rate. The findings of this research emphasize the necessity of robust defenses against adversarial attacks in software vulnerability detection.
Towards the Terminator Economy: Assessing Job Exposure to AI through LLMs
Colombo, Emilio, Mercorio, Fabio, Mezzanzanica, Mario, Serino, Antonio
The spread and rapid development of AI-related technologies are influencing many aspects of our daily lives, from social to educational, including the labour market. Many researchers have been highlighting the key role AI and technologies play in reshaping jobs and their related tasks, either by automating or enhancing human capabilities in the workplace. Can we estimate if, and to what extent, jobs and related tasks are exposed to the risk of being automatized by state-of-the-art AI-related technologies? Our work tackles this question through a data-driven approach: (i) developing a reproducible framework that exploits a battery of open-source Large Language Models to assess current AI and robotics' capabilities in performing job-related tasks; (ii) formalising and computing an AI exposure measure by occupation, namely the teai (Task Exposure to AI) index. Our results show that about one-third of U.S. employment is highly exposed to AI, primarily in high-skill jobs (aka, white collars). This exposure correlates positively with employment and wage growth from 2019 to 2023, indicating a beneficial impact of AI on productivity. The source codes and results are publicly available, enabling the whole community to benchmark and track AI and technology capabilities over time.
Open Source AI Has Founders--and the FTC--Buzzing
Y Combinator is famed for its Demo Days, where portfolio companies pitch their apps and wares in hopes of growing from a fledgling company into the next AirBnB. But on Thursday, the startup incubator hosted a mรฉlange of founders, venture capitalists, and US policy makers in its airy industrial space in San Francisco to tackle a defining topic for so many startups today: AI as the latest frontier in the battle between Big Tech and the little guys. For many early-stage tech entrepreneurs, questions around AI can carry existential weight. Ever since ChatGPT was unleashed in late 2022, OpenAI's technology, along with fast follows from Google's and Microsoft's AI teams, has dominated the conversation around this new era of artificial intelligence. But the increasing availability--and potency--of open source AI models has the potential to upend those dynamics.
Apple agrees to stick by Biden administration's voluntary AI safeguards
Apple has joined several other tech companies in agreeing to abide by voluntary AI safeguards laid out by the Biden administration. Those who make the pledge have committed to abide by eight guidelines related to safety, security and social responsibility, including flagging societal risks such as biases; testing for vulnerabilities, watermarking AI-generated images and audio; and sharing trust and safety details with the government and other companies. Amazon, Google, Microsoft and OpenAI were among the initial adoptees of the pact, which the White House announced last July. The voluntary agreement, which is not enforceable, will expire after Congress passes laws to regulate AI. Since the guidelines were announced, Apple unveiled a suite of AI-powered features under the umbrella name of Apple Intelligence.