We develop a hierarchical controller for head-to-head autonomous racing. We first introduce a formulation of a racing game with realistic safety and fairness rules. A high-level planner approximates the original formulation as a discrete game with simplified state, control, and dynamics to easily encode the complex safety and fairness rules and calculates a series of target waypoints. The low-level controller takes the resulting waypoints as a reference trajectory and computes high-resolution control inputs by solving an alternative formulation approximation with simplified objectives and constraints. We consider two approaches for the low-level planner, constructing two hierarchical controllers. One approach uses multi-agent reinforcement learning (MARL), and the other solves a linear-quadratic Nash game (LQNG) to produce control inputs. The controllers are compared against three baselines: an end-to-end MARL controller, a MARL controller tracking a fixed racing line, and an LQNG controller tracking a fixed racing line. Quantitative results show that the proposed hierarchical methods outperform their respective baseline methods in terms of head-to-head race wins and abiding by the rules. The hierarchical controller using MARL for low-level control consistently outperformed all other methods by winning over 90% of head-to-head races and more consistently adhered to the complex racing rules. Qualitatively, we observe the proposed controllers mimicking actions performed by expert human drivers such as shielding/blocking, overtaking, and long-term planning for delayed advantages. We show that hierarchical planning for game-theoretic reasoning produces competitive behavior even when challenged with complex rules and constraints.

Drawing on differing notions of core-periphery structure From a broad perspective, innovation is understood as a form of from [21] and [2], we distinguish decentralized core-periphery, collective problem-solving. For this and other reasons, the process centralized core-periphery, and affinity network structure. We generate of innovation is understood as a social process, as social collectives networks of these three classes from stochastic block models are capable of, and in some cases optimized for, both retaining (SBMs), and use them to run an agent-based model (ABM) of collective the knowledge of previous generations while building upon this cultural innovation, in which agents can only directly interact knowledge for subsequent innovations, a phenemonon we refer to with their network neighbors. In order to discover the highestscoring as "cumulative" culture [18, 19]. Human social networks tend to innovation, agents must discover and combine the highest exhibit core-periphery structures, whereby a'core' population is innovations from two completely parallel technology trees. We find heavily inter-connected, and connected in turn to more'peripheral' that decentralized core-periphery networks outperform both centralized individuals and subcommunities [2]. Prior work on the structure core-periphery networks and affinity networks, in terms of of human networks has suggested that innovation emerges at the mean crossover time for this final innovation. We hypothesize that boundary between the core and periphery of creative networks [4, decentralized core-periphery network structure provides a more 6]. Individual innovators are often in an intermediate position with fruitful environment for collective problem-solving, by allowing many core and peripheral connections, and successfully innovative for the relative shielding of periphery nodes from the optimal innovations teams tend to include both core and peripheral individuals [4].

High-Dimensional and Incomplete matrices, which usually contain a large amount of valuable latent information, can be well represented by a Latent Factor Analysis model. The performance of an LFA model heavily rely on its optimization process. Thereby, some prior studies employ the Particle Swarm Optimization to enhance an LFA model's optimization process. However, the particles within the swarm follow the static evolution paths and only share the global best information, which limits the particles' searching area to cause sub-optimum issue. To address this issue, this paper proposes a Dynamic-neighbor-cooperated Hierarchical PSO-enhanced LFA model with two-fold main ideas. First is the neighbor-cooperated strategy, which enhances the randomly chosen neighbor's velocity for particles' evolution. Second is the dynamic hyper-parameter tunning. Extensive experiments on two benchmark datasets are conducted to evaluate the proposed DHPL model. The results substantiate that DHPL achieves a higher accuracy without hyper-parameters tunning than the existing PSO-incorporated LFA models in representing an HDI matrix.

Enhancing the diversity of policies is beneficial for robustness, exploration, and transfer in reinforcement learning (RL). In this paper, we aim to seek diverse policies in an under-explored setting, namely RL tasks with structured action spaces with the two properties of composability and local dependencies. The complex action structure, non-uniform reward landscape, and subtle hyperparameter tuning due to the properties of structured actions prevent existing approaches from scaling well. We propose a simple and effective RL method, Diverse Policy Optimization (DPO), to model the policies in structured action space as the energy-based models (EBM) by following the probabilistic RL framework. A recently proposed novel and powerful generative model, GFlowNet, is introduced as the efficient, diverse EBM-based policy sampler. DPO follows a joint optimization framework: the outer layer uses the diverse policies sampled by the GFlowNet to update the EBM-based policies, which supports the GFlowNet training in the inner layer. Experiments on ATSC and Battle benchmarks demonstrate that DPO can efficiently discover surprisingly diverse policies in challenging scenarios and substantially outperform existing state-of-the-art methods.

The problem of traffic light control is to coordinate between intersections by controlling their traffic lights to improve traffic flow. This problem remains as one of the greatest challenges in the 21 st century (Qadri, Gökçe, & Öner, 2020). To tackle this challenge, researchers have taken various approaches such as the coordinated method modifying the start time of the green lights between the consecutive intersections (Koonce & Rodegerdts, 2008), the optimization technique minimizing the vehicles' travel time under certain traffic flow assumptions (Diakaki, Papageorgiou, & Aboudolas, 2002), and the models applying perimeter control to handle transferring flows between regions of a city (Kouvelas, Saeedmanesh, & Geroliminis, 2015, 2017). In addition to conventional approaches, the problem was recently tackled with Reinforcement Learning (RL) methods (Qadri et al., 2020). RL is a promising machinelearning framework where an agent interacts within a given environment by applying actions and receiving signals, which are interpreted as rewards and punishments. Via the interactions, the agents learn an optimal policy, a probability distribution over the available actions that maximizes the total obtained rewards for each visited environment state (Alamiyan-Harandi, Derhami, & Jamshidi, 2018; Rasheed, Yau, Noor, Wu, & Low, 2020; Sutton, Barto, et al., 1998). Encompassing several intersections, the traffic light control problem requires several actions to be executed at the same time. Hence, often the Multi-Agent (MA) extension of RL, i.e., MARL, is used for this problem.

Digging out the latent information from large-scale incomplete matrices is a key issue with challenges. The Latent Factor Analysis (LFA) model has been investigated in depth to an alyze the latent information. Recently, Swarm Intelligence-related LFA models have been proposed and adopted widely to improve the optimization process of LFA with high efficiency, i.e., the Particle Swarm Optimization (PSO)-LFA model. However, the hyper-parameters of the PSO-LFA model have to tune manually, which is inconvenient for widely adoption and limits the learning rate as a fixed value. To address this issue, we propose an Adam-enhanced Hierarchical PSO-LFA model, which refines the latent factors with a sequential Adam-adjusting hyper-parameters PSO algorithm. First, we design the Adam incremental vector for a particle and construct the Adam-enhanced evolution process for particles. Second, we refine all the latent factors of the target matrix sequentially with our proposed Adam-enhanced PSO's process. The experimental results on four real datasets demonstrate that our proposed model achieves higher prediction accuracy with its peers.

Despite the dramatic success in image generation, Generative Adversarial Networks (GANs) still face great challenges in synthesizing sequences of discrete elements, in particular human language. The difficulty in generator training arises from the limited representation capacity and uninformative learning signals obtained from the discriminator. In this work, we (1) first empirically show that the mixture-of-experts approach is able to enhance the representation capacity of the generator for language GANs and (2) harness the Feature Statistics Alignment (FSA) paradigm to render fine-grained learning signals to advance the generator training. Specifically, FSA forces the mean statistics of the distribution of fake data to approach that of real samples as close as possible in the finite-dimensional feature space. Empirical study on synthetic and real benchmarks shows the superior performance in quantitative evaluation and demonstrates the effectiveness of our approach to adversarial text generation.

This work considers multiple agents traversing a network from a source node to the goal node. The cost to an agent for traveling a link has a private as well as a congestion component. The agent's objective is to find a path to the goal node with minimum overall cost in a decentralized way. We model this as a fully decentralized multi-agent reinforcement learning problem and propose a novel multi-agent congestion cost minimization (MACCM) algorithm. Our MACCM algorithm uses linear function approximations of transition probabilities and the global cost function. In the absence of a central controller and to preserve privacy, agents communicate the cost function parameters to their neighbors via a time-varying communication network. Moreover, each agent maintains its estimate of the global state-action value, which is updated via a multi-agent extended value iteration (MAEVI) sub-routine. We show that our MACCM algorithm achieves a sub-linear regret. The proof requires the convergence of cost function parameters, the MAEVI algorithm, and analysis of the regret bounds induced by the MAEVI triggering condition for each agent. We implement our algorithm on a two node network with multiple links to validate it. We first identify the optimal policy, the optimal number of agents going to the goal node in each period. We observe that the average regret is close to zero for 2 and 3 agents. The optimal policy captures the trade-off between the minimum cost of staying at a node and the congestion cost of going to the goal node. Our work is a generalization of learning the stochastic shortest path problem.

Agent-based computational economics is a field with a rich academic history, yet one which has struggled to enter mainstream policy design toolboxes--plagued by the challenges associated with representing a complex and dynamic reality. The field of Reinforcement Learning (RL), too, has a rich history, and has recently been at the centre of several exponential developments. Modern RL implementations have been able to achieve unprecedented levels of sophistication, handling previously unthinkable degrees of complexity. This review surveys the historical barriers of classical agent-based techniques in economic modelling, and contemplates whether recent developments in RL can overcome any of them.

A critical factor in utilizing agents with Artificial Intelligence (AI) is their robustness to novelty. AI agents include models that are either engineered or trained. Engineered models include knowledge of those aspects of the environment that are known and considered important by the engineers. Learned models form embeddings of aspects of the environment based on connections made through the training data. In operation, however, a rich environment is likely to present challenges not seen in training sets or accounted for in engineered models. Worse still, adversarial environments are subject to change by opponents. A program at the Defense Advanced Research Project Agency (DARPA) seeks to develop the science necessary to develop and evaluate agents that are robust to novelty. This capability will be required, before AI has the role envisioned within mission critical environments. As part of the Science of AI and Learning for Open-world Novelty (SAIL-ON), we are mapping possible military domain novelty types to a domain-independent ontology developed as part of a theory of novelty. Characterizing the possible space of novelty mathematically and ontologically will allow us to experiment with agent designs that are coming from the DARPA SAIL-ON program in relevant military environments. Utilizing the same techniques as being used in laboratory experiments, we will be able to measure agent ability to detect, characterize, and accommodate novelty.