DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models

Personal use of this material is permitted. Abstract--Human drivers possess spatial and causal intelligence, enabling them to perceive driving scenarios, anticipate hazards, and react to dynamic environments. In contrast, autonomous vehicles lack these abilities, making it challenging to manage perception-related Safety of the Intended Functionality (SOTIF) risks, especially under complex or unpredictable driving conditions. T o address this gap, we propose fine-tuning multimodal large language models (MLLMs) on a customized dataset specifically designed to capture perception-related SOTIF scenarios. Benchmarking results show that fine-tuned MLLMs achieve an 11.8% improvement in close-ended VQA accuracy and a 12.0% increase in open-ended VQA scores compared to baseline models, while maintaining real-time performance with a 0.59-second average inference time per image. We validate our approach through real-world case studies in Canada and China, where fine-tuned models correctly identify safety risks that challenge even experienced human drivers. This work represents the first application of domain-specific MLLM fine-tuning for the SOTIF domain in autonomous driving. N autonomous driving (AD), safety is commonly classified into functional safety and Safety of the Intended Functionality (SOTIF). Functional safety concerns failures in hardware or software that result in unsafe operation. In contrast, SOTIF addresses hazards that occur not due to malfunctions, but when the system operates as intended yet produces unsafe outcomes because of external factors or inherent limitations [1]. Perception systems in autonomous vehicles (A Vs), which are tasked with detecting, classifying, and predicting based on environmental stimuli, are particularly vulnerable to SOTIF-related challenges. Manuscript received 2 February, 2025; revised 27 August, 2025; accepted 7 September, 2025. Y ang, and A. Khajepour are with MVS-Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, 200 University Ave West, Waterloo ON, N2L3G1 Canada. S. Huang, and F. Shi are with CompLING Lab, David R. Cheriton School of Computer Science, University of Waterloo, 200 University Ave West, Waterloo ON, N2L3G1 Canada and V ector Institute, Toronto, Canada C. Sun is with the Department of Data and Systems Engineering, University of Hong Kong, Pok Fu Lam, Hong Kong, China (e-mail: c87sun@hku.hk) Lu is with the Department of Mechanical Engineering, University of New Brunswick, Fredericton, NB E3B 5A3, Canada (e-mail: yukun.lu@unb.ca) H. Wang is with School of V ehicle and Mobility, Tsinghua University, Beijing, China, 100084.