Touch-based Curiosity for Sparse-Reward Tasks

Abstract--Robots in many real-world settings have access to force/torque sensors in their gripper and tactile sensing is often necessary in tasks that involve contact-rich motion. In this work, we leverage surprise from mismatches in touch feedback to guide exploration in hard sparse-reward reinforcement learning tasks. Our approach, Touch-based Curiosity (ToC), learns what visible objects interactions are supposed to "feel" like. We encourage exploration by rewarding interactions where the expectation and the experience don't match. In our proposed method, an initial task-independent exploration phase is followed by an on-task learning phase, in which the original interactions are relabeled with on-task rewards. We test our approach on a range of touchintensive robot arm tasks (e.g. In the former, the environment is often fully observable, and the reward is dense and well-defined. In the Recent works in RL have focused on curiosity-driven latter, a large amount of work is required to design useful exploration through prediction-based surprise [6, 45, 48]. While it may be possible to hand-craft dense formulation, a forward dynamics models predicts the future, and reward signals for many real-world tasks, we believe that it's if its prediction is incorrect when compared to the real future, a worthwhile endeavor to investigate learning methods that do the agent is surprised and is thus rewarded.