Performance Analysis
RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors
Eddoubi, Hicham, Ricker, Jonas, Cocchi, Federico, Baraldi, Lorenzo, Sotgiu, Angelo, Pintor, Maura, Cornia, Marcella, Baraldi, Lorenzo, Fischer, Asja, Cucchiara, Rita, Biggio, Battista
AI-generated images have reached a quality level at which humans are incapable of reliably distinguishing them from real images. To counteract the inherent risk of fraud and disinformation, the detection of AI-generated images is a pressing challenge and an active research topic. While many of the presented methods claim to achieve high detection accuracy, they are usually evaluated under idealized conditions. In particular, the adversarial robustness is often neglected, potentially due to a lack of awareness or the substantial effort required to conduct a comprehensive robustness analysis. In this work, we tackle this problem by providing a simpler means to assess the robustness of AI-generated image detectors. We present RAID (Robust evaluation of AI-generated image Detectors), a dataset of 72k diverse and highly transferable adversarial examples. The dataset is created by running attacks against an ensemble of seven state-of-the-art detectors and images generated by four different text-to-image models. Extensive experiments show that our methodology generates adversarial images that transfer with a high success rate to unseen detectors, which can be used to quickly provide an approximate yet still reliable estimate of a detector's adversarial robustness. Our findings indicate that current state-of-the-art AI-generated image detectors can be easily deceived by adversarial examples, highlighting the critical need for the development of more robust methods. We release our dataset at https://huggingface.co/datasets/aimagelab/RAID and evaluation code at https://github.com/pralab/RAID.
Protap: A Benchmark for Protein Modeling on Realistic Downstream Applications
Yan, Shuo, Yan, Yuliang, Ma, Bin, Li, Chenao, Tang, Haochun, Lu, Jiahua, Lin, Minhua, Feng, Yuyuan, Xiong, Hui, Dai, Enyan
Recently, extensive deep learning architectures and pretraining strategies have been explored to support downstream protein applications. Additionally, domain-specific models incorporating biological knowledge have been developed to enhance performance in specialized tasks. In this work, we introduce $\textbf{Protap}$, a comprehensive benchmark that systematically compares backbone architectures, pretraining strategies, and domain-specific models across diverse and realistic downstream protein applications. Specifically, Protap covers five applications: three general tasks and two novel specialized tasks, i.e., enzyme-catalyzed protein cleavage site prediction and targeted protein degradation, which are industrially relevant yet missing from existing benchmarks. For each application, Protap compares various domain-specific models and general architectures under multiple pretraining settings. Our empirical studies imply that: (i) Though large-scale pretraining encoders achieve great results, they often underperform supervised encoders trained on small downstream training sets. (ii) Incorporating structural information during downstream fine-tuning can match or even outperform protein language models pretrained on large-scale sequence corpora. (iii) Domain-specific biological priors can enhance performance on specialized downstream tasks. Code and datasets are publicly available at https://github.com/Trust-App-AI-Lab/protap.
Amatriciana: Exploiting Temporal GNNs for Robust and Efficient Money Laundering Detection
Di Gennaro, Marco, Panebianco, Francesco, Pianta, Marco, Zanero, Stefano, Carminati, Michele
Money laundering is a financial crime that poses a serious threat to financial integrity and social security. The growing number of transactions makes it necessary to use automatic tools that help law enforcement agencies detect such criminal activity. In this work, we present Amatriciana, a novel approach based on Graph Neural Networks to detect money launderers inside a graph of transactions by considering temporal information. Amatriciana uses the whole graph of transactions without splitting it into several time-based subgraphs, exploiting all relational information in the dataset. Our experiments on a public dataset reveal that the model can learn from a limited amount of data. Furthermore, when more data is available, the model outperforms other State-of-the-art approaches; in particular, Amatriciana decreases the number of False Positives (FPs) while detecting many launderers. In summary, Amatriciana achieves an F1 score of 0.76. In addition, it lowers the FPs by 55% with respect to other State-of-the-art models.
Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models
Xiong, Guangzhi, Xie, Eric, Williams, Corey, Kim, Myles, Shariatmadari, Amir Hassan, Guo, Sikun, Bekiranov, Stefan, Zhang, Aidong
Large language models (LLMs) have shown significant potential in scientific disciplines such as biomedicine, particularly in hypothesis generation, where they can analyze vast literature, identify patterns, and suggest research directions. However, a key challenge lies in evaluating the truthfulness of generated hypotheses, as verifying their accuracy often requires substantial time and resources. Additionally, the hallucination problem in LLMs can lead to the generation of hypotheses that appear plausible but are ultimately incorrect, undermining their reliability. To facilitate the systematic study of these challenges, we introduce Truth-Hypo, a benchmark for assessing the capabilities of LLMs in generating truthful scientific hypotheses, and KnowHD, a knowledge-based hallucination detector to evaluate how well hypotheses are grounded in existing knowledge. Our results show that LLMs struggle to generate truthful hypotheses. By analyzing hallucinations in reasoning steps, we demonstrate that the groundedness scores provided by KnowHD serve as an effective metric for filtering truthful hypotheses from the diverse outputs of LLMs.
Benchmarking Early Agitation Prediction in Community-Dwelling People with Dementia Using Multimodal Sensors and Machine Learning
Abedi, Ali, Chu, Charlene H., Khan, Shehroz S.
Agitation is one of the most common responsive behaviors in people living with dementia, particularly among those residing in community settings without continuous clinical supervision. Timely prediction of agitation can enable early intervention, reduce caregiver burden, and improve the quality of life for both patients and caregivers. This study aimed to develop and benchmark machine learning approaches for the early prediction of agitation in community-dwelling older adults with dementia using multimodal sensor data. A new set of agitation-related contextual features derived from activity data was introduced and employed for agitation prediction. A wide range of machine learning and deep learning models was evaluated across multiple problem formulations, including binary classification for single-timestamp tabular sensor data and multi-timestamp sequential sensor data, as well as anomaly detection for single-timestamp tabular sensor data. The study utilized the Technology Integrated Health Management (TIHM) dataset, the largest publicly available dataset for remote monitoring of people living with dementia, comprising 2,803 days of in-home activity, physiology, and sleep data. The most effective setting involved binary classification of sensor data using the current 6-hour timestamp to predict agitation at the subsequent timestamp. Incorporating additional information, such as time of day and agitation history, further improved model performance, with the highest AUC-ROC of 0.9720 and AUC-PR of 0.4320 achieved by the light gradient boosting machine. This work presents the first comprehensive benchmarking of state-of-the-art techniques for agitation prediction in community-based dementia care using privacy-preserving sensor data. The approach enables accurate, explainable, and efficient agitation prediction, supporting proactive dementia care and aging in place.
Facial Foundational Model Advances Early Warning of Coronary Artery Disease from Live Videos with DigitalShadow
Zhou, Juexiao, Han, Zhongyi, Xin, Mankun, He, Xingwei, Wang, Guotao, Song, Jiaoyan, Luo, Gongning, He, Wenjia, Li, Xintong, Chu, Yuetan, Chen, Juanwen, Wang, Bo, Wu, Xia, Duan, Wenwen, Guo, Zhixia, Bai, Liyan, Pan, Yilin, Bi, Xuefei, Liu, Lu, Feng, Long, He, Xiaonan, Gao, Xin
Abstract--Global population aging presents increasing challenges to healthcare systems, with coronary artery disease (CAD) responsible for approximately 17.8 million deaths annually, making it a leading cause of global mortality . As CAD is largely preventable, early detection and proactive management are essential. In this work, we introduce DigitalShadow, an advanced early warning system for CAD, powered by a fine-tuned facial foundation model. The system is pre-trained on 21 million facial images and subsequently fine-tuned into LiveCAD, a specialized CAD risk assessment model trained on 7,004 facial images from 1,751 subjects across four hospitals in China. DigitalShadow functions passively and contactlessly, extracting facial features from live video streams without requiring active user engagement. Integrated with a personalized database, it generates natural language risk reports and individualized health recommendations. With privacy as a core design principle, DigitalShadow supports local deployment to ensure secure handling of user data. The world's population is rapidly ageing [1], with significant implications for the prevalence of chronic diseases such as Coronary Artery Disease (CAD) [2], affecting not only individuals but also families and societies at large [3]. The number of older people is increasing at an unprecedented rate, projected to grow from approximately 761 million in 2021 to 1.6 billion by 2050, which would represent nearly 16% of the global population, according to the UN's W orld Social Report 2023 [4]. The aging population presents numerous challenges, including increased pressure on healthcare systems, pension schemes, and long-term care facilities, alongside potential economic consequences, which together fuel growing demand for healthcare services [5], [6]. With advancing age, people become more vulnerable to various critical diseases [7], such as CAD [8], stroke [9], cancer [10], and Parkinson's disease (PD) [11], [12], [13], leading to considerable morbidity and mortality [14].
Nonlinear Causal Discovery through a Sequential Edge Orientation Approach
Recent advances have established the identifiability of a directed acyclic graph (DAG) under additive noise models (ANMs), spurring the development of various causal discovery methods. However, most existing methods make restrictive model assumptions, rely heavily on general independence tests, or require substantial computational time. To address these limitations, we propose a sequential procedure to orient undirected edges in a completed partial DAG (CPDAG), representing an equivalence class of DAGs, by leveraging the pairwise additive noise model (PANM) to identify their causal directions. We prove that this procedure can recover the true causal DAG assuming a restricted ANM. Building on this result, we develop a novel constraint-based algorithm for learning causal DAGs under nonlinear ANMs. Given an estimated CPDAG, we develop a ranking procedure that sorts undirected edges by their adherence to the PANM, which defines an evaluation order of the edges. To determine the edge direction, we devise a statistical test that compares the log-likelihood values, evaluated with respect to the competing directions, of a sub-graph comprising just the candidate nodes and their identified parents in the partial DAG. We further establish the structural learning consistency of our algorithm in the large-sample limit. Extensive experiments on synthetic and real-world datasets demonstrate that our method is computationally efficient, robust to model misspecification, and consistently outperforms many existing nonlinear DAG learning methods.
Neural Responses to Affective Sentences Reveal Signatures of Depression
Kommineni, Aditya, Jeong, Woojae, Avramidis, Kleanthis, McDaniel, Colin, Hughes, Myzelle, McGee, Thomas, Kaiser, Elsi, Lerman, Kristina, Blank, Idan A., Byrd, Dani, Habibi, Assal, Cahn, B. Rael, Kadiri, Sudarsana, Medani, Takfarinas, Leahy, Richard M., Narayanan, Shrikanth
Depression is one of the most prevalent mental health disorders worldwide, with estimates indicating that around 5% of the worlds' adult population [1, 2] suffers from this condition. The primary methods for screening and monitoring depression rely on self-reported questionnaires, such as the Patient Health Questionnaire (PHQ-9) [3], Beck's Depression Inventory (BDI) [4] and Hamilton Depression Ratings Scale (HDRS) [5]. While these questionnaires are effective to varying degrees at screening patients for depression, they provide only limited information about the affected underlying neuro-cognitive processes in individuals, limiting the ability to personalize treatments. Given the heterogeneity of depressive symptomatology across patient populations [6, 7], it is crucial to elucidate the underlying neurophysiological mechanisms to support the development of more effective and individualized procedures for screening, monitoring, and treatment. Prior functional imaging studies have identified increased activity in anterior cin-gulate cortex (especially the subgenual anterior cingulate) during presentation of emotional stimuli, altered connectivity in prefrontal cortical areas, and default mode network as potential differentiating markers in depressed participants [8-13].
Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completeness
Wang, Cheng-Long, Li, Qi, Xiang, Zihang, Cao, Yinzhi, Wang, Di
Growing concerns over data privacy and security highlight the importance of machine unlearning--removing specific data influences from trained models without full retraining. Techniques like Membership Inference Attacks (MIAs) are widely used to externally assess successful unlearning. However, existing methods face two key limitations: (1) maximizing MIA effectiveness (e.g., via online attacks) requires prohibitive computational resources, often exceeding retraining costs; (2) MIAs, designed for binary inclusion tests, struggle to capture granular changes in approximate unlearning. To address these challenges, we propose the Interpolated Approximate Measurement (IAM), a framework natively designed for unlearning inference. IAM quantifies sample-level unlearning completeness by interpolating the model's generalization-fitting behavior gap on queried samples. IAM achieves strong performance in binary inclusion tests for exact unlearning and high correlation for approximate unlearning--scalable to LLMs using just one pre-trained shadow model. We theoretically analyze how IAM's scoring mechanism maintains performance efficiently. We then apply IAM to recent approximate unlearning algorithms, revealing general risks of both over-unlearning and under-unlearning, underscoring the need for stronger safeguards in approximate unlearning systems. The code is available at https://github.com/Happy2Git/Unlearning_Inference_IAM.
Hey, That's My Data! Label-Only Dataset Inference in Large Language Models
Xiong, Chen, Wang, Zihao, Zhu, Rui, Ho, Tsung-Yi, Chen, Pin-Yu, Xiong, Jingwei, Tang, Haixu, Ohno-Machado, Lucila
Large Language Models (LLMs) have revolutionized Natural Language Processing by excelling at interpreting, reasoning about, and generating human language. However, their reliance on large-scale, often proprietary datasets poses a critical challenge: unauthorized usage of such data can lead to copyright infringement and significant financial harm. Existing dataset-inference methods typically depend on log probabilities to detect suspicious training material, yet many leading LLMs have begun withholding or obfuscating these signals. This reality underscores the pressing need for label-only approaches capable of identifying dataset membership without relying on internal model logits. We address this gap by introducing CatShift, a label-only dataset-inference framework that capitalizes on catastrophic forgetting: the tendency of an LLM to overwrite previously learned knowledge when exposed to new data. If a suspicious dataset was previously seen by the model, fine-tuning on a portion of it triggers a pronounced post-tuning shift in the model's outputs; conversely, truly novel data elicits more modest changes. By comparing the model's output shifts for a suspicious dataset against those for a known non-member validation set, we statistically determine whether the suspicious set is likely to have been part of the model's original training corpus. Extensive experiments on both open-source and API-based LLMs validate CatShift's effectiveness in logit-inaccessible settings, offering a robust and practical solution for safeguarding proprietary data.