Recent advances in multiagent learning have seen the introduction ofa family of algorithms that revolve around the population-based trainingmethod PSRO, showing convergence to Nash, correlated and coarse corre-lated equilibria. Notably, when the number of agents increases, learningbest-responses becomes exponentially more difficult, and as such ham-pers PSRO training methods. The paradigm of mean-field games pro-vides an asymptotic solution to this problem when the considered gamesare anonymous-symmetric. Unfortunately, the mean-field approximationintroduces non-linearities which prevent a straightforward adaptation ofPSRO. Building upon optimization and adversarial regret minimization,this paper sidesteps this issue and introduces mean-field PSRO, an adap-tation of PSRO which learns Nash, coarse correlated and correlated equi-libria in mean-field games. The key is to replace the exact distributioncomputation step by newly-defined mean-field no-adversarial-regret learn-ers, or by black-box optimization. We compare the asymptotic complexityof the approach to standard PSRO, greatly improve empirical bandit con-vergence speed by compressing temporal mixture weights, and ensure itis theoretically robust to payoff noise. Finally, we illustrate the speed andaccuracy of mean-field PSRO on several mean-field games, demonstratingconvergence to strong and weak equilibria.

With the increasing ubiquity of autonomous robotic solutions, the interest in their connectivity and in the cooperation within multi-robot systems is rising. Two aspects that are a matter of current research are robot security and secure multi-robot collaboration robust to byzantine agents. Blockchain and other distributed ledger technologies (DLTs) have been proposed to address the challenges in both domains. Nonetheless, some key challenges include scalability and deployment within real-world networks. This paper presents an approach to integrating IOTA and ROS 2 for more scalable DLT-based robotic systems while allowing for network partition tolerance after deployment. This is, to the best of our knowledge, the first implementation of IOTA smart contracts for robotic systems, and the first integrated design with ROS 2. This is in comparison to the vast majority of the literature which relies on Ethereum. We present a general IOTA+ROS 2 architecture leading to partition-tolerant decision-making processes that also inherit byzantine tolerance properties from the embedded blockchain structures. We demonstrate the effectiveness of the proposed framework for a cooperative mapping application in a system with intermittent network connectivity. We show both superior performance with respect to Ethereum in the presence of network partitions, and a low impact in terms of computational resource utilization. These results open the path for wider integration of blockchain solutions in distributed robotic systems with less stringent connectivity and computational requirements.

Conflict-Based Search (CBS) is a popular multi-agent path finding (MAPF) solver that employs a low-level single agent planner and a high-level constraint tree to resolve conflicts. The vast majority of modern MAPF solvers focus on improving CBS by reducing the size of this tree through various strategies with few methods modifying the low level planner. Typically low level planners in existing CBS methods use an unweighted cost-to-go heuristic, with suboptimal CBS methods also using a conflict heuristic to help the high level search. In this paper, we show that, contrary to prevailing CBS beliefs, a weighted cost-to-go heuristic can be used effectively alongside the conflict heuristic in two possible variants. In particular, one of these variants can obtain large speedups, 2-100x, across several scenarios and suboptimal CBS methods. Importantly, we discover that performance is related not to the weighted cost-to-go heuristic but rather to the relative conflict heuristic weight's ability to effectively balance low-level and high-level work, implying that existing suboptimal CBS work misses this subtlety. Additionally, to the best of our knowledge, we show the first theoretical relation of prioritized planning and bounded suboptimal CBS and demonstrate that our methods are their natural generalization.

We consider communication in a fully cooperative multi-agent system, where the agents have partial observation of the environment and must act jointly to maximize the overall reward. We have a discrete-time queueing network where agents route packets to queues based only on the partial information of the current queue lengths. The queues have limited buffer capacity, so packet drops happen when they are sent to a full queue. In this work, we implemented a communication channel for the agents to share their information in order to reduce the packet drop rate. For efficient information sharing we use an attention-based communication model, called ATVC, to select informative messages from other agents. The agents then infer the state of queues using a combination of the variational auto-encoder, VAE, and product-of-experts, PoE, model. Ultimately, the agents learn what they need to communicate and with whom, instead of communicating all the time with everyone. We also show empirically that ATVC is able to infer the true state of the queues and leads to a policy which outperforms existing baselines.

Systems for relative localization in multi-robot systems based on ultra-wideband (UWB) ranging have recently emerged as robust solutions for GNSS-denied environments. Scalability remains one of the key challenges, particularly in ad-hoc deployments. Recent solutions include dynamic allocation of active and passive localization modes for different robots or nodes in the system. With larger-scale systems becoming more distributed, key research questions arise in the areas of security and trustability of such localization systems. This paper studies the potential integration of collaborative-decision making processes with distributed ledger technologies. Specifically, we investigate the design and implementation of a methodology for running an UWB role allocation algorithm within smart contracts in a blockchain. In previous works, we have separately studied the integration of ROS2 with the Hyperledger Fabric blockchain, and introduced a new algorithm for scalable UWB-based localization. In this paper, we extend these works by (i) running experiments with larger number of mobile robots switching between different spatial configurations and (ii) integrating the dynamic UWB role allocation algorithm into Fabric smart contracts for distributed decision-making in a system of multiple mobile robots. This enables us to deliver the same functionality within a secure and trustable process, with enhanced identity and data access management. Our results show the effectiveness of the UWB role allocation for continuously varying spatial formations of six autonomous mobile robots, while demonstrating a low impact on latency and computational resources of adding the blockchain layer that does not affect the localization process.

A major challenge in multi-agent systems is that the system complexity grows dramatically with the number of agents as well as the size of their action spaces, which is typical in real world scenarios such as autonomous vehicles, robotic teams, network routing, etc. It is hence in imminent need to design decentralized or independent algorithms where the update of each agent is only based on their local observations without the need of introducing complex communication/coordination mechanisms. In this work, we study the finite-time convergence of independent entropy-regularized natural policy gradient (NPG) methods for potential games, where the difference in an agent's utility function due to unilateral deviation matches exactly that of a common potential function. The proposed entropy-regularized NPG method enables each agent to deploy symmetric, decentralized, and multiplicative updates according to its own payoff. We show that the proposed method converges to the quantal response equilibrium (QRE) -- the equilibrium to the entropy-regularized game -- at a sublinear rate, which is independent of the size of the action space and grows at most sublinearly with the number of agents. Appealingly, the convergence rate further becomes independent with the number of agents for the important special case of identical-interest games, leading to the first method that converges at a dimension-free rate. Our approach can be used as a smoothing technique to find an approximate Nash equilibrium (NE) of the unregularized problem without assuming that stationary policies are isolated.

Reinforcement learning (RL) often requires decomposing a problem into subtasks and composing learned behaviors on these tasks. Compositionality in RL has the potential to create modular subtask units that interface with other system capabilities. However, generating compositional models requires the characterization of minimal assumptions for the robustness of the compositional feature. We develop a framework for a \emph{compositional theory} of RL using a categorical point of view. Given the categorical representation of compositionality, we investigate sufficient conditions under which learning-by-parts results in the same optimal policy as learning on the whole. In particular, our approach introduces a category $\mathsf{MDP}$, whose objects are Markov decision processes (MDPs) acting as models of tasks. We show that $\mathsf{MDP}$ admits natural compositional operations, such as certain fiber products and pushouts. These operations make explicit compositional phenomena in RL and unify existing constructions, such as puncturing hazardous states in composite MDPs and incorporating state-action symmetry. We also model sequential task completion by introducing the language of zig-zag diagrams that is an immediate application of the pushout operation in $\mathsf{MDP}$.

Commonsense (background) knowledge, at least the kind of knowledge that we fetch and relay upon in the process of language understanding: (i) cannot be learned by processing vast amounts of text because that knowledge is never explicitly stated in the text -- and you cannot find what's not there; and (ii) that background knowledge cannot be learned perceptually from observation since the vast amount of the crucial background knowledge is universal, is not probablistic nor approximate, and so it cannot be susceptible to individual observations. The shared background knowledge needed in the process of language understanding is the kind of knowledge that obeys and respects the laws of nature and as such it has to be codified. In fact, that knowledge must be codified in a symbolic system that quantifies over variables of specific ontological types. There's a consensus among researchers investigating the neurological, psychological and evolutionary aspects of human linguistic communication that languages have evolved according to the information-theoretic principle of least effort. Specifically, it has been established that interacting communicative agents tend to produce utterances that minimize the complexity of coding a thought as well as minimize the process of decoding linguistic utterances back to the intended thought [1] -- thus finding an optimal point where the effort of both speaker and listener is minimal.

Mixed integer convex and nonlinear programs, MICP and MINLP, are expressive but require long solving times. Recent work that combines data-driven methods on solver heuristics has shown potential to overcome this issue allowing for applications on larger scale practical problems. To solve mixed-integer bilinear programs online with data-driven methods, several formulations exist including mathematical programming with complementary constraints (MPCC), mixed-integer programming (MIP). In this work, we benchmark the performances of those data-driven schemes on a bookshelf organization problem that has discrete mode switch and collision avoidance constraints. The success rate, optimal cost and solving time are compared along with non-data-driven methods. Our proposed methods are demonstrated as a high level planner for a robotic arm for the bookshelf problem.

Unmanned aerial vehicles (UAVs) are being employed in many areas such as photography, emergency, entertainment, defence, agriculture, forestry, mining and construction. Over the last decade, UAV technology has found applications in numerous construction project phases, ranging from site mapping, progress monitoring, building inspection, damage assessments, and material delivery. While extensive studies have been conducted on the advantages of UAVs for various construction-related processes, studies on UAV collaboration to improve the task capacity and efficiency are still scarce. This paper proposes a new cooperative path planning algorithm for multiple UAVs based on the stag hunt game and particle swarm optimization (PSO). First, a cost function for each UAV is defined, incorporating multiple objectives and constraints. The UAV game framework is then developed to formulate the multi-UAV path planning into the problem of finding payoff-dominant equilibrium. Next, a PSO-based algorithm is proposed to obtain optimal paths for the UAVs. Simulation results for a large construction site inspected by three UAVs indicate the effectiveness of the proposed algorithm in generating feasible and efficient flight paths for UAV formation during the inspection task.