ConcertoRL: An Innovative Time-Interleaved Reinforcement Learning Approach for Enhanced Control in Direct-Drive Tandem-Wing Vehicles

In control problems for insect-scale direct-drive experimental platforms under tandem wing influence, the precision and safety of control during plug-and-play online training and control processes are paramount. The primary challenge facing existing reinforcement learning models is their limited safety in the exploration process and the stability of the continuous training process. Addressing these challenges, we introduce the ConcertoRL algorithm to enhance control precision and stabilize the online training process, which consists of two main innovations: a time-interleaved mechanism to interweave classical controllers with reinforcement learning-based controllers aiming to improve control precision in the initial stages, a policy composer organizes the experience gained from previous learning to ensure the stability of the online training process. This paper conducts a series of experiments. First, experiments incorporating the time-interleaved mechanism demonstrate a substantial performance boost of approximately 70% over scenarios without reinforcement learning enhancements and a 50% increase in efficiency compared to reference controllers with doubled control frequencies. These results highlight the algorithm's ability to create a synergistic effect that exceeds the sum of its parts. Second, Ablation studies on the policy composer further reveal that this module significantly enhances the stability of ConcertoRL during online training. Lastly, experiments on the universality of the current ConcertoRL algorithm framework demonstrate its compatibility with various classical controllers, consistently achieving excellent control outcomes. ConcertoRL sets a new benchmark in control effectiveness for challenges posed by direct-drive platforms under tandem wing influence and establishes a comprehensive framework for integrating classical and reinforcement learning-based control methodologies.