Sim-to-Real Transfer in Deep Reinforcement Learning for Bipedal Locomotion

Abstract--This chapter addresses the critical challenge of simulation-to-reality (sim-to-real) transfer for deep reinforcement learning (DRL) in bipedal locomotion. The first is to shrink the gap through model-centric strategies that systematically improve the simulator's physical fidelity. The second is to harden the policy, a complementary approach that uses in-simulation robustness training and post-deployment adaptation to make the policy inherently resilient to model inaccuracies. The chapter concludes by synthesizing these philosophies into a strategic framework, providing a clear roadmap for developing and evaluating robust sim-to-real solutions. Bipedal robots, machines that walk on two legs, are compelling platforms for operation in human-centric and natural environments. They can climb stairs, step over irregular obstacles, traverse narrow passages, and access spaces that are impractical for wheeled platforms. Their anthropomorphic form factor also enables natural interaction with tools and infrastructure designed for humans, making them suitable for disaster response, healthcare, logistics, and industrial applications. Bipedal locomotion remains challenging because of its high dimensionality, underactuation, and intermittent contacts. Model-based methods struggle with complex dynamics, whereas deep reinforcement learning (DRL) has achieved impressive simulation results in bipedal locomotion through trial and error. As shown in Figure 1, DRL achieves more robust performance than model-based control, particularly as task complexity increases. Most controllers adopt either end-to-end policies that map observations to actions or hierarchical policies that decouple high-level (HL) intent from low-level (LL) execution. Both approaches perform well in simulation but transfer unreliably to hardware, a limitation known as the sim-to-real gap.