PillarAcc: Sparse PointPillars Accelerator for Real-Time Point Cloud 3D Object Detection on Edge Devices

PointPillars, a widely adopted bird's-eye view (BEV) encoding, aggregates 3D point cloud data into 2D pillars for high-accuracy 3D object detection. However, most state-of-the-art methods employing PointPillar overlook the inherent sparsity of pillar encoding, missing opportunities for significant computational reduction. In this study, we propose a groundbreaking algorithm-hardware co-design that accelerates sparse convolution processing and maximizes sparsity utilization in pillar-based 3D object detection networks. We investigate sparsification opportunities using an advanced pillar-pruning method, achieving an optimal balance between accuracy and sparsity. We introduce PillarAcc, a state-ofthe-art sparsity support mechanism that enhances sparse pillar convolution through linear complexity input-output mapping generation and conflict-free gather-scatter memory access. Additionally, we propose dataflow optimization techniques, Figure 1: Challenges in PointPillars acceleration and improvements dynamically adjusting the pillar processing schedule by this work: (a) up to three orders of magnitude for optimal hardware utilization under diverse sparsity increase in frames per second at equivalent accuracy by proposed operations. We evaluate PillarAcc on various cutting-edge PillarAcc, (b) degraded sparsity across layers by convolution 3D object detection networks and benchmarks, achieving (conv.) vs. maintained sparsity (this work), (c) significant remarkable speedup and energy savings compared to representative sparsity mapping overhead in conventional system edge platforms, demonstrating record-breaking (conv.) vs. reduced mapping overhead and enhanced computing PointPillars speed of 500FPS with minimal compromise in efficiency (this work).