Optimizing Energy Management of Smart Grid using Reinforcement Learning aided by Surrogate models built using Physics-informed Neural Networks

Optimizing the energy management within a smart grids scenario presents significant challenges, primarily due to the complexity of real-world systems and the intricate interactions among various components. Reinforcement Learning (RL) is gaining prominence as a solution for addressing the challenges of Optimal Power Flow (OPF) in smart grids. However, RL needs to iterate compulsively throughout a given environment to obtain the optimal policy. This means obtaining samples from a, most likely, costly simulator, which can lead to a sample e fficiency problem. In this work, we address this problem by substituting costly smart grid simulators with surrogate models built using Physics-Informed Neural Networks (PINN)s, optimizing the RL policy training process by arriving to convergent results in a fraction of the time employed by the original environment. Specifically, we tested the performance of our PINN surrogate against other state-of-the-art data-driven surrogates and found that the understanding of the underlying physical nature of the problem makes the PINN surrogate the only method that we studied capable of learning a good RL policy, in addition to not having to use samples from the real simulator. Our work shows that, by employing PINN surrogates, we can improve training speed by 50%, comparing to training the RL policy by not using any surrogate model, enabling us to achieve results with score on par with the original simulator more rapidly. Keywords: Smart Grids Control, Reinforcement Learning, Physics-informed Neural Networks, Active Network Management, Optimal Power Flow, Surrogate Models, Renewable EnergyRL Reinforcement Learning EA Expert agent PINN Physics-Informed Neural Networks ANN Artificial Neural Network OPF Optimal Power Flow ESS Energy Storage Systems SoC State of Change MAE Mean Absolute Error 1. Introduction Smart grids are a pivotal concept driving the current modernization of electrical networks, addressing the urgent need to reduce greenhouse gas emissions, enhance energy e fficiency, and improve grid stability through demand response mechanisms. The European Union aims to achieve 43% renewable energy generation by 2030 [1], and in 2021, the renewable energy share rose to 32 .1% [2]. Corresponding author Email address: julen.cestero@polimi.it Modern societies require advanced grids capable of predicting and mitigating the uncertainties associated with renewable energy sources.