Imitation learning methods need significant human supervision to learn policies robust to changes in object poses, physical disturbances, and visual distractors. Reinforcement learning, on the other hand, can explore the environment autonomously to learn robust behaviors but may require impractical amounts of unsafe real-world data collection. To learn performant, robust policies without the burden of unsafe real-world data collection or extensive human supervision, we propose RialTo, a system for robustifying real-world imitation learning policies via reinforcement learning in "digital twin" simulation environments constructed on the fly from small amounts of real-world data. To enable this real-to-sim-to-real pipeline, RialTo proposes an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. We also introduce a novel "inverse distillation" procedure for bringing real-world demonstrations into simulated environments for efficient fine-tuning, with minimal human intervention and engineering required. We evaluate RialTo across a variety of robotic manipulation problems in the real world, such as robustly stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases (over 67%) in policy robustness without requiring extensive human data collection. Project website and videos at https://real-to-sim-to-real.github.io/RialTo/

This morning, before checking in on my young son or making a coffee, I opened the Duolingo app on my phone and translated "They love smelling meat" into Italian. I've been starting my days like this for a few months now: wake up, wash face, grapple with the gerund. I usually spend between 10 and 20 minutes on it while the kettle boils or I load CBeebies or write some emails. Duolingo is a language learning app and pretty simple to use. After you've chosen which language you want to learn, you are presented with about 100 skill-sets divided by scenario or grammar (grocery shopping, the future tense and so on).