Multi-Agent Reinforcement Learning (MARL) is an increasingly important research field that can model and control multiple large-scale autonomous systems. Despite its achievements, existing multi-agent learning methods typically involve expensive computations in terms of training time and power arising from large observation-action space and a huge number of training steps. Therefore, a key challenge is understanding and characterizing the computationally intensive functions in several popular classes of MARL algorithms during their training phases. Our preliminary experiments reveal new insights into the key modules of MARL algorithms that limit the adoption of MARL in real-world systems. We explore neighbor sampling strategy to improve cache locality and observe performance improvement ranging from 26.66% (3 agents) to 27.39% (12 agents) during the computationally intensive mini-batch sampling phase. Additionally, we demonstrate that improving the locality leads to an end-to-end training time reduction of 10.2% (for 12 agents) compared to existing multi-agent algorithms without significant degradation in the mean reward.

Abstract--Multi-Agent Reinforcement Learning (MARL) is a promising area of research that can model and control multiple, autonomous decision-making agents. During online training, MARL algorithms involve performance-intensive computations such as exploration and exploitation phases originating from large observation-action space belonging to multiple agents. In this article, we seek to characterize the scalability bottlenecks in several popular classes of MARL algorithms during their training phases. Our experimental results reveal new insights into the key modules of MARL algorithms that limit the scalability, and outline potential strategies that may help address these performance issues. Distributed Artificial Intelligence < I.2 Artificial Intelligence < I Computing Methodologies The function that determines the action is known as a policy.

You must be thinking, another Backprop from scratch blog? Well kinda yes but I thought this through and came up with something that you can use to tinker around along with easy to understand equations that you usually write down to understand the algorithm. This blog will focus on implementing the Backpropagation algorithm step-by-step on mini-batches of the dataset. There are plenty of tutorials and blogs to demonstrate the backpropagation algorithm in detail and all the logic behind calculus and algebra happening. So I'll skip that part and cut to equations in math and implementation using Python (coz why not).