Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions

Non-prehensile manipulation is a crucial skill for enabling versatile robots to interact with ungraspable objects, using actions such as pushing, rolling, or tossing. However, achieving dexterous non-prehensile manipulation in robots poses significant challenges. During contact interactions, different contact modes arise such as sticking, sliding, and separation, and transitions between these contact modes lead to hybrid dynamics [1, 2, 3]. Furthermore, due to its underactuated nature, it requires long-term reasoning about contact interactions as well as reactive control to recover from mistakes and disturbances [1, 2]. The frictional interactions between the robot, the object, and the environment are difficult to model, which creates uncertainty in the behavior of the object [4, 5]. The highly uncertain nature of the underactuated frictional interactions [4, 5] make the nonprehensile manipulation problem especially sensitive to occlusions. Previous non-prehensile works assume near-perfect visual perception from external systems, providing either point-cloud [6] or pose observations [7, 8, 9, 10, 11]. However, moving towards more versatile onboard perception will make frequent occlusions unavoidable, either due to obstacles in the environment, self occlusions, or even human-induced occlusions, for instance in a human-robot collaboration setting. In this paper, we propose a learning-based system for non-prehensile manipulation that leverages tactile sensing to overcome occlusions in the visual perception.