Goto

Collaborating Authors

 Instructional Material


CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Models

arXiv.org Artificial Intelligence

Ensuring safety in autonomous driving systems remains a critical challenge, particularly in handling rare but potentially catastrophic safety-critical scenarios. While existing research has explored generating safety-critical scenarios for autonomous vehicle (AV) testing, there is limited work on effectively incorporating these scenarios into policy learning to enhance safety. Furthermore, developing training curricula that adapt to an AV's evolving behavioral patterns and performance bottlenecks remains largely unexplored. To address these challenges, we propose CurricuVLM, a novel framework that leverages Vision-Language Models (VLMs) to enable personalized curriculum learning for autonomous driving agents. Our approach uniquely exploits VLMs' multimodal understanding capabilities to analyze agent behavior, identify performance weaknesses, and dynamically generate tailored training scenarios for curriculum adaptation. Through comprehensive analysis of unsafe driving situations with narrative descriptions, CurricuVLM performs in-depth reasoning to evaluate the AV's capabilities and identify critical behavioral patterns. The framework then synthesizes customized training scenarios targeting these identified limitations, enabling effective and personalized curriculum learning. Extensive experiments on the Waymo Open Motion Dataset show that CurricuVLM outperforms state-of-the-art baselines across both regular and safety-critical scenarios, achieving superior performance in terms of navigation success, driving efficiency, and safety metrics. Further analysis reveals that CurricuVLM serves as a general approach that can be integrated with various RL algorithms to enhance autonomous driving systems. The code and demo video are available at: https://zihaosheng.github.io/CurricuVLM/.


LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models

arXiv.org Artificial Intelligence

Existing Large Vision-Language Models (LVLMs) can process inputs with context lengths up to 128k visual and text tokens, yet they struggle to generate coherent outputs beyond 1,000 words. We find that the primary limitation is the absence of long output examples during supervised fine-tuning (SFT). To tackle this issue, we introduce LongWriter-V-22k, a SFT dataset comprising 22,158 examples, each with multiple input images, an instruction, and corresponding outputs ranging from 0 to 10,000 words. Moreover, to achieve long outputs that maintain high-fidelity to the input images, we employ Direct Preference Optimization (DPO) to the SFT model. Given the high cost of collecting human feedback for lengthy outputs (e.g., 3,000 words), we propose IterDPO, which breaks long outputs into segments and uses iterative corrections to form preference pairs with the original outputs. Additionally, we develop MMLongBench-Write, a benchmark featuring six tasks to evaluate the long-generation capabilities of VLMs. Our 7B parameter model, trained with LongWriter-V-22k and IterDPO, achieves impressive performance on this benchmark, outperforming larger proprietary models like GPT-4o. Code and data: https://github.com/THU-KEG/LongWriter-V


ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification

arXiv.org Artificial Intelligence

Self-awareness, i.e., the ability to assess and correct one's own generation, is a fundamental aspect of human intelligence, making its replication in large language models (LLMs) an important yet challenging task. Previous works tackle this by employing extensive reinforcement learning or rather relying on large external verifiers. In this work, we propose Refine via Intrinsic Self-Verification (ReVISE), an efficient and effective framework that enables LLMs to self-correct their outputs through self-verification. The core idea of ReVISE is to enable LLMs to verify their reasoning processes and continually rethink reasoning trajectories based on its verification. We introduce a structured curriculum based upon online preference learning to implement this efficiently. Specifically, as ReVISE involves two challenging tasks (i.e., self-verification and reasoning correction), we tackle each task sequentially using curriculum learning, collecting both failed and successful reasoning paths to construct preference pairs for efficient training. During inference, our approach enjoys natural test-time scaling by integrating self-verification and correction capabilities, further enhanced by our proposed confidence-aware decoding mechanism. Our experiments on various reasoning tasks demonstrate that ReVISE achieves efficient self-correction and significantly improves reasoning performance.


What's coming up at #AAAI2025?

AIHub

From Tuesday 25 February to Tuesday 4 March 2025, Philadelphia will play host to the 39th Annual AAAI Conference on Artificial Intelligence (AAAI 2025). The event will feature invited talks, tutorials, workshops, and an extensive technical programme. There are also a whole host of other sessions, including a doctoral consortium, diversity and inclusion activities, posters, demos, and more. We (AIhub) will be running a science communication training session on Wednesday 26 February. There are eight invited talks this year.


Personalized Education with Generative AI and Digital Twins: VR, RAG, and Zero-Shot Sentiment Analysis for Industry 4.0 Workforce Development

arXiv.org Artificial Intelligence

While the advent of the Fourth Industrial Revolution (4IR) technologies, like cloud computing, machine learning, and artificial intelligence have brought convenience and productivity improvements, they have also introduced new challenges in training and education that require the reskilling of existing employees and the building of a new workforce. Exacerbated by the already existing workforce shortages, this mammoth workforce reskilling and building effort aims to build a high-tech workforce capable of operating and maintaining these 4IR systems; requiring a higher student retention and persistence. This increase in student retention and persistence will be especially critical when training the workforce originating from marginalized communities like Underrepresented Minorities (URM), where challenges arise due to lack of access to high-quality education throughout the trainee's formative years (pre/middle/high schools), creating a cyclic set of knowledge dependencies that are difficult to meet. To address these challenges, this research presents Generative AI-based Personalized Tutor for Industrial 4.0 (gAI-PT4I4), a framework that focuses on personalization of 4IR experiential learning, using sentiment analysis to gauge student's knowledge comprehension, while using a combination of generative AI and finite automaton to personalize the content to the students' learning needs. The framework administers experiential learning, using low-fidelity Digital Twins that enable virtual reality-based (VR) training exercises focusing on 4IR training. The VR environment, integrates a generative AI teaching assistant called the Interactive Tutor, that guides the student through the training exercises, with audio and text communications.


Conformal Prediction as Bayesian Quadrature

arXiv.org Machine Learning

As machine learning-based prediction systems are increasingly used in high-stakes situations, it is important to understand how such predictive models will perform upon deployment. Distribution-free uncertainty quantification techniques such as conformal prediction provide guarantees about the loss black-box models will incur even when the details of the models are hidden. However, such methods are based on frequentist probability, which unduly limits their applicability. We revisit the central aspects of conformal prediction from a Bayesian perspective and thereby illuminate the shortcomings of frequentist guarantees. We propose a practical alternative based on Bayesian quadrature that provides interpretable guarantees and offers a richer representation of the likely range of losses to be observed at test time.


Flow-based generative models as iterative algorithms in probability space

arXiv.org Machine Learning

Generative AI (GenAI) has revolutionized data-driven modeling by enabling the synthesis of high-dimensional data across various applications, including image generation, language modeling, biomedical signal processing, and anomaly detection. Flow-based generative models provide a powerful framework for capturing complex probability distributions, offering exact likelihood estimation, efficient sampling, and deterministic transformations between distributions. These models leverage invertible mappings governed by Ordinary Differential Equations (ODEs), enabling precise density estimation and likelihood evaluation. This tutorial presents an intuitive mathematical framework for flow-based generative models, formulating them as neural network-based representations of continuous probability densities. We explore key theoretical principles, including the Wasserstein metric, gradient flows, and density evolution governed by ODEs, to establish convergence guarantees and bridge empirical advancements with theoretical insights. By providing a rigorous yet accessible treatment, we aim to equip researchers and practitioners with the necessary tools to effectively apply flow-based generative models in signal processing and machine learning.


You need to MIMIC to get FAME: Solving Meeting Transcript Scarcity with a Multi-Agent Conversations

arXiv.org Artificial Intelligence

Meeting summarization suffers from limited high-quality data, mainly due to privacy restrictions and expensive collection processes. We address this gap with FAME, a dataset of 500 meetings in English and 300 in German produced by MIMIC, our new multi-agent meeting synthesis framework that generates meeting transcripts on a given knowledge source by defining psychologically grounded participant profiles, outlining the conversation, and orchestrating a large language model (LLM) debate. A modular post-processing step refines these outputs, mitigating potential repetitiveness and overly formal tones, ensuring coherent, credible dialogues at scale. We also propose a psychologically grounded evaluation framework assessing naturalness, social behavior authenticity, and transcript difficulties. Human assessments show that FAME approximates real-meeting spontaneity (4.5/5 in naturalness), preserves speaker-centric challenges (3/5 in spoken language), and introduces richer information-oriented difficulty (4/5 in difficulty). These findings highlight that FAME is a good and scalable proxy for real-world meeting conditions. It enables new test scenarios for meeting summarization research and other conversation-centric applications in tasks requiring conversation data or simulating social scenarios under behavioral constraints.


This 30 bundle helps you write, plan & create easier with ChatGPT

PCWorld

Artificial intelligence has changed the way we write, create, work, and plan--and with the 2025 ChatGPT Skills & Creativity Bundle, you'll be at the forefront of this revolution. For a limited time, you can access five expert-led courses that teach you how to supercharge your creativity, productivity, and AI skills for just 29.99 instead of 249. Whether you're an aspiring content creator, business professional, or someone who simply wants to make life easier with AI, this bundle will equip you with practical skills to get the most out of ChatGPT. From crafting engaging blog posts and video scripts to enhancing office productivity, bulk content creation, and even travel planning, this bundle offers real-world applications that will help you streamline your work and spark your creativity. The flagship course, Creative Writing & Content Creation with ChatGPT, guides you through advanced prompt engineering techniques to generate high-quality, engaging content while maintaining your unique voice.


Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

arXiv.org Artificial Intelligence

We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. We scale a proof-of-concept model to 3.5 billion parameters and 800 billion tokens. We show that the resulting model can improve its performance on reasoning benchmarks, sometimes dramatically, up to a computation load equivalent to 50 billion parameters.