DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation

Our system incorporates several key components, including a Finite Element Method (FEM)-based soft body model for simulating the sensing elastomer, a multi-material simulator for modeling diverse object types (such as elastic, elastoplastic, cables) under manipulation, a penalty-based contact model for handling contact dynamics. Additionally, we introduce a method to infer the optical response of our tactile sensor to contact using an efficient pixel-based neural module. In the goal of enabling robots to perform human-level manipulation on a diverse set of tasks, touch is one of the most prominent components. Tactile sensing, as a modality, is unique in the sense that it provides accurate, fine-detailed information about environmental interactions in the form of contact geometries and forces. Although its efficacy has been highlighted by prior research, providing crucial feedback in grasping fragile objects (Ishikawa et al., 2022), enabling robots to perform in occluded environment (Yu & Rodriguez, 2018), and detecting incipient slip (Chen et al., 2018) for highly reactive grasping, there are still advances in tactile sensing to be made especially in the form of simulation. Physics-based simulation has become a significant practical tool in the domain of robotics, by mitigating the challenges of real-world design and verification of learning algorithms. This work was done during an internship at the MIT-IBM Watson AI Lab. To accurately simulate tactile sensors which are inherently soft, it is essential to model soft body interaction's contact geometries, forces, and dynamics. Prior work (Si & Yuan, 2022) attempted to simulate contact geometries and forces for tactile sensors under (quasi-)static scenarios, and it was successfully applied to robotic perception tasks such as object shape estimation (Suresh et al., 2022), and grasp stability prediction (Si et al., 2022). However, highly dynamic manipulation tasks have not been thoroughly explored. Other prior works approach contact dynamics by either approximating sensor surface deformation using rigid-body dynamics (Xu et al., 2023) or using physics-based soft-body simulation methods such as Finite Element Method (FEM) (Narang et al., 2021). However, these methods are still limited to manipulating rigid objects.