Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions

Figure 1: Text2Touch improves upon previous reward function design methods to increase the performance of robotic in-hand object rotation in rotation speed and grasp stability. We evaluate the performance of LLM-generated reward functions using only tactile and proprioceptive information in the real world. Designing reinforcement learning (RL) reward functions for dexterous in-hand manipulation remains a formidable challenge. Traditional approaches often rely on domain experts to painstakingly specify and tune reward terms [1], a process prone to suboptimal or unintended behaviours [2, 3]. Recent work has shown that large language models (LLMs) can generate policy or reward code for robotic tasks [4, 5, 6, 7, 8, 9], a notable step toward reducing manual engineering. However, these breakthroughs have primarily focused on conventional sensing modalities (vision, proprioception) for real-world validation [4, 10, 6, 7, 8]. To date, tactile sensing has not yet been integrated into automated reward generation via LLMs in either simulated or real-world settings. Vision based tactile sensing can provide detailed contact and force signals that visual sensing alone often fails to capture, especially under occlusions or subtle slip conditions [11].