Distributed Power Control for Large Energy Harvesting Networks: A Multi-Agent Deep Reinforcement Learning Approach

In this paper, we develop a multi-agent reinforcement learning (MARL) framework to obtain online power control policies for a large energy harvesting (EH) multiple access channel, when only the causal information about the EH process and wireless channel is available. In the proposed framework, we model the online power control problem as a discrete-time mean-field game (MFG), and leverage the deep reinforcement learning technique to learn the stationary solution of the game in a distributed fashion. We analytically show that the proposed procedure converges to the unique stationary solution of the MFG. Using the proposed framework, the power control policies are learned in a completely distributed fashion. In order to benchmark the performance of the distributed policies, we also develop a deep neural network (DNN) based centralized as well as distributed online power control schemes. Our simulation results show the efficacy of the proposed power control policies. In particular, the DNN based centralized power control policies provide a very good performance for large EH networks for which the design of optimal policies is intractable using the conventional methods such as Markov decision processes. Further, performance of both the distributed policies is close to the throughput achieved by the centralized policies. The work in this paper will appear in part at IEEE ICASSP 2019 [1] and IEEE WiOpt 2019 [2]. This research has been partly supported by the ERC-PoC 727682 CacheMire project. I. INTRODUCTION Internet-of-things (IoT) [3] networks connect a large number of low power sensors whose lifespan is typically limited by the energy that can be stored in their batteries. In this context, the advent of the energy harvesting (EH) technology [4] promises to prolong the lifespan of IoT networks by enabling the nodes to operate by harvesting energy from environmental sources, e.g., the sun, the wind, etc.