Towards Latency-Aware 3D Streaming Perception for Autonomous Driving

T owards Latency-aware 3D Streaming Perception for Autonomous Driving Jiaqi Peng 1,2, Tai Wang 2, Jiangmiao Pang 2 and Y uan Shen 1,2 Abstract -- Although existing 3D perception algorithms have demonstrated significant improvements in performance, their deployment on edge devices continues to encounter critical challenges due to substantial runtime latency. We propose a new benchmark tailored for online evaluation by considering runtime latency. Based on the benchmark, we build a Latency-A ware 3D Streaming Perception (LASP) framework that addresses the latency issue through two primary components: 1) latency-aware history integration, which extends query propagation into a continuous process, ensuring the integration of historical feature regardless of varying latency; 2) latency-aware predictive detection, a module that compensates the detection results with the predicted trajectory and the posterior accessed latency. By incorporating the latency-aware mechanism, our method shows generalization across various latency levels, achieving an online performance that closely aligns with 80% of its offline evaluation on the Jetson AGX Orin without any acceleration techniques. I. INTRODUCTION 3D perception is an essential capability for autonomous vehicles and provides the foundation for subsequent prediction and planning [1], [2]. The past few years have witnessed the rapid advancement of 3D perception algorithms [3]- [5]. In particular, one of the most popular settings, i.e. [6], [7], only using multiple cameras, can achieve performance that is comparable with LiDAR-based methods [8], [9] with effective BEV -based paradigms [5], [6].