Hierarchical Deep Deterministic Policy Gradient for Autonomous Maze Navigation of Mobile Robots

Maze navigation is a fundamental challenge in robotics, requiring agents to traverse complex environments efficiently. While the Deep Deterministic Policy Gradient (DDPG) algorithm excels in control tasks, its performance in maze navigation suffers from sparse rewards, inefficient exploration, and long-horizon planning difficulties, often leading to low success rates and average rewards, sometimes even failing to achieve effective navigation. To address these limitations, this paper proposes an efficient Hierarchical DDPG (HDDPG) algorithm, which includes high-level and low-level policies. The high-level policy employs an advanced DDPG framework to generate intermediate subgoals from a long-term perspective and on a higher temporal scale. The low-level policy, also powered by the improved DDPG algorithm, generates primitive actions by observing current states and following the subgoal assigned by the high-level policy. The proposed method enhances stability with off-policy correction, refining subgoal assignments by relabeling historical experiences. Additionally, adaptive parameter space noise is utilized to improve exploration, and a reshaped intrinsic-extrinsic reward function is employed to boost learning efficiency. Further optimizations, including gradient clipping and Xavier initialization, are employed to improve robustness. The proposed algorithm is rigorously evaluated through numerical simulation experiments executed using the Robot Operating System (ROS) and Gazebo. Regarding the three distinct final targets in autonomous maze navigation tasks, HDDPG significantly overcomes the limitations of standard DDPG and its variants, improving the success rate by at least 56.59% and boosting the average reward by a minimum of 519.03 compared to baseline algorithms. Keywords: Hierarchical Deep Reinforcement Learning; Deep Deterministic Policy Gradient; Off-policy Correction; Adaptive Parameter Space Noise; Reshaping Reward Function; Autonomous Maze Navigation1. Introduction In recent years, autonomous navigation for mobile robots has emerged as a pivotal research focus, driven by its potential to enhance efficiency and safety across diverse applications, which mainly strive to enable mobile robots to reach the target point while effectively avoiding obstacles [1, 2]. Maze navigation, as one of the diverse autonomous navigation tasks, poses distinct challenges. This problem is particularly relevant in critical scenarios such as warehouse logistics, search and rescue operations in disaster-stricken areas, and autonomous exploration of hazardous or unknown environments, where precise and adaptive path planning is crucial for success.