Characterization of Human Balance through a Reinforcement Learning-based Muscle Controller

Abstract: Balance assessment during physical rehabilitation often relies on rubricoriented battery tests to score a patient's physical capabilities, leading to subjectivity. While some objective balance assessments exist, they are often limited to tracking the center of pressure (COP), which does not fully capture the whole-body postural stability. This study explores the use of the center of mass (COM) state space and presents a promising avenue for monitoring the balance capabilities in humans. We employ a musculoskeletal model integrated with a balance controller, trained through reinforcement learning (RL), to investigate balancing capabilities. The RL framework consists of two interconnected neural networks governing balance recovery and muscle coordination respectively, trained using Proximal Policy Optimization (PPO) with reference state initialization, early termination, and multiple training strategies. By exploring recovery from random initial COM states (position and velocity) space for a trained controller, we obtain the final BR enclosing successful balance recovery trajectories. Comparing the BRs with analytical postural stability limits from a linear inverted pendulum model, we observe a similar trend in successful COM states but more limited ranges in the recoverable areas. We further investigate the effect of muscle weakness and neural excitation delay on the BRs, revealing reduced balancing capability in different regions. Overall, our approach of learning muscular balance controllers presents a promising new method for establishing balance recovery limits and objectively assessing balance capability in bipedal systems, particularly in humans. Keywords: Balance, Reinforcement Learning, Musculoskeletal Modeling, Bipedal Systems, Motor Disorders 1. Introduction Falls and subsequent injuries pose a significant health risk for the elderly and mobility-impaired populations. Poor balancing capabilities are the leading cause of falls in the elderly population, which reduces the overall quality of life of aging patients [1-3]. The injuries sustained by these patients can range from lower-body fractures, particularly in the hip, to head injuries, with falls being the leading cause of traumatic brain injuries [4]. Therefore, effective balance assessment and rehabilitation are critical components not only to health monitoring and injury prevention in mobility-impaired individuals, but also to the diagnoses of other serious underlying medical conditions. Since balance is maintained through a complicated network of physiological systems in the body, it is difficult to pinpoint a single origin causing deficiencies in patients and to assess balance through simple isolated measures. In most clinical environments, balance assessment is performed as a battery of balance exercises designed to evaluate the patient's ability to perform selected tasks.