Material-informed Gaussian Splatting for 3D World Reconstruction in a Digital Twin

Abstract--3D reconstruction for Digital Twins often relies on LiDAR-based methods, which provide accurate geometry but lack the semantics and textures naturally captured by cameras. Traditional LiDAR-camera fusion approaches require complex calibration and still struggle with certain materials like glass, which are visible in images but poorly represented in point clouds. We propose a camera-only pipeline that reconstructs scenes using 3D Gaussian Splatting from multi-view images, extracts semantic material masks via vision models, converts Gaussian representations to mesh surfaces with projected material labels, and assigns physics-based material properties for accurate sensor simulation in modern graphics engines and simulators. This approach combines photorealistic reconstruction with physics-based material assignment, providing sensor simulation fidelity comparable to LiDAR-camera fusion while eliminating hardware complexity and calibration requirements. Modern Advanced Driver Assistance Systems (ADAS) and Artificial Intelligence (AI) systems rely on sophisticated algorithms and complex multimodal sensor setups, often including LiDAR sensors and cameras. Achieving accurate translation of real-world scenarios to simulation--where virtual sensors exhibit identical behavior to their physical counterparts--requires 3D world reconstruction that captures spatial and geometric features to ensure realistic simulation and synthetic sensor data consistent with reality. Previous LiDAR-based approaches [36, 17] achieve accurate geometry but lack camera-captured texture information. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HADEA).