Law
Comprehensive Framework for Evaluating Conversational AI Chatbots
Gupta, Shailja, Ranjan, Rajesh, Singh, Surya Narayan
Conversational AI chatbots are transforming industries by streamlining customer service, automating transactions, and enhancing user engagement. However, evaluating these systems remains a challenge, particularly in financial services, where compliance, user trust, and operational efficiency are critical. This paper introduces a novel evaluation framework that systematically assesses chatbots across four dimensions: cognitive and conversational intelligence, user experience, operational efficiency, and ethical and regulatory compliance. By integrating advanced AI methodologies with financial regulations, the framework bridges theoretical foundations and real-world deployment challenges. Additionally, we outline future research directions, emphasizing improvements in conversational coherence, real-time adaptability, and fairness.
CSEval: Towards Automated, Multi-Dimensional, and Reference-Free Counterspeech Evaluation using Auto-Calibrated LLMs
Hengle, Amey, Kumar, Aswini, Bandhakavi, Anil, Chakraborty, Tanmoy
Counterspeech has emerged as a popular and effective strategy for combating online hate speech, sparking growing research interest in automating its generation using language models. However, the field still lacks standardised evaluation protocols and reliable automated evaluation metrics that align with human judgement. Current automatic evaluation methods, primarily based on similarity metrics, do not effectively capture the complex and independent attributes of counterspeech quality, such as contextual relevance, aggressiveness, or argumentative coherence. This has led to an increased dependency on labor-intensive human evaluations to assess automated counter-speech generation methods. To address these challenges, we introduce CSEval, a novel dataset and framework for evaluating counterspeech quality across four dimensions: contextual-relevance, aggressiveness, argument-coherence, and suitableness. Furthermore, we propose Auto-Calibrated COT for Counterspeech Evaluation (Auto-CSEval), a prompt-based method with auto-calibrated chain-of-thoughts (CoT) for scoring counterspeech using large language models. Our experiments show that Auto-CSEval outperforms traditional metrics like ROUGE, METEOR, and BertScore in correlating with human judgement, indicating a significant improvement in automated counterspeech evaluation.
YINYANG-ALIGN: Benchmarking Contradictory Objectives and Proposing Multi-Objective Optimization based DPO for Text-to-Image Alignment
Das, Amitava, Narsupalli, Yaswanth, Singh, Gurpreet, Jain, Vinija, Sharma, Vasu, Trivedy, Suranjana, Chadha, Aman, Sheth, Amit
Precise alignment in Text-to-Image (T2I) systems is crucial to ensure that generated visuals not only accurately encapsulate user intents but also conform to stringent ethical and aesthetic benchmarks. Incidents like the Google Gemini fiasco, where misaligned outputs triggered significant public backlash, underscore the critical need for robust alignment mechanisms. In contrast, Large Language Models (LLMs) have achieved notable success in alignment. Building on these advancements, researchers are eager to apply similar alignment techniques, such as Direct Preference Optimization (DPO), to T2I systems to enhance image generation fidelity and reliability. We present YinYangAlign, an advanced benchmarking framework that systematically quantifies the alignment fidelity of T2I systems, addressing six fundamental and inherently contradictory design objectives. Each pair represents fundamental tensions in image generation, such as balancing adherence to user prompts with creative modifications or maintaining diversity alongside visual coherence. YinYangAlign includes detailed axiom datasets featuring human prompts, aligned (chosen) responses, misaligned (rejected) AI-generated outputs, and explanations of the underlying contradictions.
Generative Adversarial Networks Bridging Art and Machine Intelligence
Song, Junhao, Zhang, Yichao, Bi, Ziqian, Wang, Tianyang, Chen, Keyu, Li, Ming, Niu, Qian, Liu, Junyu, Peng, Benji, Zhang, Sen, Liu, Ming, Xu, Jiawei, Pan, Xuanhe, Wang, Jinlang, Feng, Pohsun, Wen, Yizhu, Yan, Lawrence K. Q., Tseng, Hong-Ming, Song, Xinyuan, Ren, Jintao, Chen, Silin, Wang, Yunze, Hsieh, Weiche, Jing, Bowen, Yang, Junjie, Zhou, Jun, Yao, Zheyu, Liang, Chia Xin
Generative Adversarial Networks (GAN) have greatly influenced the development of computer vision and artificial intelligence in the past decade and also connected art and machine intelligence together. This book begins with a detailed introduction to the fundamental principles and historical development of GANs, contrasting them with traditional generative models and elucidating the core adversarial mechanisms through illustrative Python examples. The text systematically addresses the mathematical and theoretical underpinnings including probability theory, statistics, and game theory providing a solid framework for understanding the objectives, loss functions, and optimisation challenges inherent to GAN training. Subsequent chapters review classic variants such as Conditional GANs, DCGANs, InfoGAN, and LAPGAN before progressing to advanced training methodologies like Wasserstein GANs, GANs with gradient penalty, least squares GANs, and spectral normalisation techniques. The book further examines architectural enhancements and task-specific adaptations in generators and discriminators, showcasing practical implementations in high resolution image generation, artistic style transfer, video synthesis, text to image generation and other multimedia applications. The concluding sections offer insights into emerging research trends, including self-attention mechanisms, transformer-based generative models, and a comparative analysis with diffusion models, thus charting promising directions for future developments in both academic and applied settings.
Online Social Support Detection in Spanish Social Media Texts
Tash, Moein Shahiki, Ramos, Luis, Ahani, Zahra, Monroy, Raul, kolesnikova, Olga, Calvo, Hiram, Sidorov, Grigori
The advent of social media has transformed communication, enabling individuals to share their experiences, seek support, and participate in diverse discussions. While extensive research has focused on identifying harmful content like hate speech, the recognition and promotion of positive and supportive interactions remain largely unexplored. This study proposes an innovative approach to detecting online social support in Spanish-language social media texts. We introduce the first annotated dataset specifically created for this task, comprising 3,189 YouTube comments classified as supportive or non-supportive. To address data imbalance, we employed GPT-4o to generate paraphrased comments and create a balanced dataset. We then evaluated social support classification using traditional machine learning models, deep learning architectures, and transformer-based models, including GPT-4o, but only on the unbalanced dataset. Subsequently, we utilized a transformer model to compare the performance between the balanced and unbalanced datasets. Our findings indicate that the balanced dataset yielded improved results for Task 2 (Individual and Group) and Task 3 (Nation, Other, LGBTQ, Black Community, Women, Religion), whereas GPT-4o performed best for Task 1 (Social Support and Non-Support). This study highlights the significance of fostering a supportive online environment and lays the groundwork for future research in automated social support detection.
Decoding Complexity: Intelligent Pattern Exploration with CHPDA (Context Aware Hybrid Pattern Detection Algorithm)
Koli, Lokesh, Kalra, Shubham, Singh, Karanpreet
Efficient data management is essential for organizations to ensure that sensitive information such as Personally Identifiable Information (PII), Protected Health Information (PHI) and financial records are systematically identified and protected. Effective classification aids in compliance with regulations such as the General Data Protection Regulation (GDPR) and the Health Insurance Portability and Accountability Act (HIPAA), while mitigating security risks through real-time threat detection[3] Automated tools improve operational efficiency by streamlining access and eliminating redundancies. Customized classification systems fulfill global compliance requirements, while centralized control mechanisms enhance governance through unified policy enforcement.[4] Strategic data classification is crucial to achieve security, compliance, and operational effectiveness in the digital environment of today. Identifying PII and PHI across various data formats presents considerable challenges, particularly with unstructured data sets. Differences in encoding and file formats (e.g., PDFs, Word documents, databases, CSV, and other text files) and data storage systems complicate the consistent extraction of sensitive information [5]. Moreover, international regulations such as GDPR, HIPAA, and the California Consumer Privacy Act (CCPA) impose varied compliance mandates, adding further complexity to detection efforts. Customizing detection mechanisms to align with region-specific regulations while ensuring accuracy across different content types is formidable. The necessity for real-time detection and the reduction of false positives amplifies this challenge, necessitating advanced algorithms and comprehensive data management strategies.
Big Tech whistleblower's parents sue, sounding alarm over son's unexpected death
If you or someone you know is having thoughts of suicide, please contact the Suicide & Crisis Lifeline at 988 or 1-800-273-TALK (8255). The parents of a young California tech whistleblower whose 2024 death was ruled a suicide are now suing the City and County of San Francisco, alleging they violated public records laws by refusing to fulfill their requests for information about their son's death. Suchir Balaji, 26, was an employee at OpenAI, the artificial intelligence company behind ChatGPT, at the time of his Nov. 26, 2024, death. A San Francisco County medical examiner concluded the next day he died from a self-inflicted gunshot wound inside his apartment. "In the two-plus months since their son's passing, Petitioners and their counsel have been stymied at every turn as they have sought more information about the cause of and circumstances surrounding Suchir's tragic death. This petition, they hope, is the beginning of the end of that obstruction," the lawsuit states.
Review for NeurIPS paper: Can the Brain Do Backpropagation? --- Exact Implementation of Backpropagation in Predictive Coding Networks
Weaknesses: I have some critical remarks: 1.) Weight transport problem. This problem is not solved in the model. In fact the model needs symmetric weights. Feedback alignment will probably not work here, as I assume that the existence of an equilibrium state necessitates symmetric weights. The authors claim that the update rules are local.
Review for NeurIPS paper: Can the Brain Do Backpropagation? --- Exact Implementation of Backpropagation in Predictive Coding Networks
Following the author response, we had a long discussion. On the positive side, this is the first algorithm with local update rules that exactly simulates BP (at least asymptotically, given complete convergence at the initialization). On the negative side, all reviewers agreed this algorithm has some reduced plausibility. Specifically, in IL (original PCN) we have to present both input and output, and wait sufficient time until convergence. In contrast, in Z-IL and Fa-Z-IL, we have to first present (only) the input, also wait sufficient time until convergence, and then present the output; In addition, the learning rule becomes more complicated (through the introduction of the Phi function) and we must detect when "the change in error node is caused by feedback input" (which seems to require some global signals). This seems more complicated and less plausible then the original IL.
Group Reasoning Emission Estimation Networks
Guo, Yanming, Qian, Xiao, Credit, Kevin, Ma, Jin
Accurate greenhouse gas (GHG) emission reporting is critical for governments, businesses, and investors. However, adoption remains limited particularly among small and medium enterprises due to high implementation costs, fragmented emission factor databases, and a lack of robust sector classification methods. To address these challenges, we introduce Group Reasoning Emission Estimation Networks (GREEN), an AI-driven carbon accounting framework that standardizes enterprise-level emission estimation, constructs a large-scale benchmark dataset, and leverages a novel reasoning approach with large language models (LLMs). Specifically, we compile textual descriptions for 20,850 companies with validated North American Industry Classification System (NAICS) labels and align these with an economic model of carbon intensity factors. By reframing sector classification as an information retrieval task, we fine-tune Sentence-BERT models using a contrastive learning loss. To overcome the limitations of single-stage models in handling thousands of hierarchical categories, we propose a Group Reasoning method that ensembles LLM classifiers based on the natural NAICS ontology, decomposing the task into multiple sub-classification steps. We theoretically prove that this approach reduces classification uncertainty and computational complexity. Experiments on 1,114 NAICS categories yield state-of-the-art performance (83.68% Top-1, 91.47% Top-10 accuracy), and case studies on 20 companies report a mean absolute percentage error (MAPE) of 45.88%. The project is available at: https://huggingface.co/datasets/Yvnminc/ExioNAICS.