Continual Deep Reinforcement Learning for Decentralized Satellite Routing

This paper introduces a full solution for decentralized routing in Low Earth Orbit Satellite Constellations (LSatCs) based on continual Deep Reinforcement Learning (DRL). This requires addressing multiple challenges, including the partial knowledge at the satellites and their continuous movement, and the time-varying sources of uncertainty in the system, such as traffic, communication links, or communication buffers. We follow a multi-agent approach, where each satellite acts as an independent decision-making agent, while acquiring a limited knowledge of the environment based on the feedback received from the nearby agents. The solution is divided into two phases. First, an offline learning phase relies on decentralized decisions and a global Deep Neural Network (DNN) trained with global experiences to learn the optimal paths at each possible position and congestion level. Then, the online phase with local, on-board, and pre-trained DNNs requires continual learning to evolve with the environment, which can be done in two different ways: (1) Model anticipation, where the predictable conditions of the constellation, resulting from its orbital dynamics, are exploited by each satellite sharing local model with the next satellite; and (2) Federated Learning (FL), where each agent's model is merged first at the cluster level and then aggregated in a global Parameter Server (PS) at ground or at a geostationary orbit (GEO) satellite. The results show that, without high congestion, the proposed Multi-Agent Deep Reinforcement Learning (MA-DRL) framework achieves the same E2E performance as a shortest-path solution, but the latter assumes intensive communication overhead for real-time network-wise knowledge of the system at a centralized node, whereas ours only requires limited feedback exchange among first neighbour satellites. Moreover, the divergence of models over time is easily tackled by the synergy between anticipation, applied in short-term alignment, and FL, utilized for long-term alignment. F. Lozano-Cuadra (flozano@ic.uma.es) and B. Soret are with the Telecommunications Research Institute, University of Malaga, 29071, Malaga, Spain. I. Leyva-Mayorga and P. Popovski are with the Connectivity Section, Aalborg University, 9220 Aalborg, Denmark. The work of F. Lozano-Cuadra and B. Soret is partially funded by the European Space Agency (ESA) framework SatNEx V (prime contract no. The view expressed herein can in no way be taken to reflect the official opinion of ESA. Through the incremental adoption of the inter-satellite link (ISL), Low Earth Orbit Satellite Constellations (LSatCs) are turning into packet-based Non-Terrestrial Networks (NTN) capable of providing ubiquituous sensing, navigation, positioning, and communication services towards 6G.