The Distribution Shift Problem in Transportation Networks using Reinforcement Learning and AI

Abstract--The use of Machine Learning (ML) and Artificial Intelligence (AI) in smart transportation networks has increased significantly in the last few years. Among these ML and AI approaches, Reinforcement Learning (RL) has been shown to be a very promising approach by several authors. However, a problem with using Reinforcement Learning in Traffic Signal Control is the reliability of the trained RL agents due to the dynamically changing distribution of the input data with respect to the distribution of the data used for training. This presents a major challenge and a reliability problem for the trained network of AI agents and could have very undesirable and even detrimental consequences if a suitable solution is not found. Several researchers have tried to address this problem using different approaches. In particular, Meta Reinforcement Learning (Meta RL) promises to be an effective solution. In this paper, we evaluate and analyze a state-of-the-art Meta RL approach called MetaLight and show that, while under certain conditions MetaLight can indeed lead to reasonably good results, under some other conditions it might not perform well (with errors of up to 22%), suggesting that Meta RL schemes are often not robust enough and can even pose major reliability problems. As cities become more populated and the number of vehicles on their roads increases, the problem of efficiently controlling the flow of vehicles to reduce travel times and CO2 emissions is becoming a top priority. For this reason, in recent years, research in Traffic Signal Control has gained significant momentum as the quest to develop better Traffic Signal Control algorithms intensified. Specifically, Deep Reinforcement Learning (Deep RL) gained much attention in the research community as it better captures the sequential decision-making nature of the problem.