Methods for combining Machine Learning (ML) and Constrained Optimization (CO) for decision support have attracted considerable interest in recent years. This is motivated by the possibility to tackle complex decision making problems subject to uncertainty (sometimes over multiple stages), and having a partially specified structure where knowledge is available both in explicit form (cost function, constraints) and implicit form (historical data or simulators). As a practical example, an Energy Management Systems (EMS) needs to allocate minimum-cost power flows from different Distributed Energy Resources (DERs) [1]. Based on actual energy prices, and forecasts on the availability of DERs and on consumption, the EMS decides which power generators should be used and whether the surplus should be stored or sold to the market. Such a problem involves hard constraints (maintaining power balance, power flow limits), a clear cost structure, elements of uncertainty that are partially known via historical data, and multiple decision stages likely subject to execution time restrictions. In this type of use case, pure CO methods struggle with robustness and scalability, while pure ML methods such as Reinforcement Learning (RL) have trouble dealing with hard constraints and combinatorial decision spaces. Motivated by the opportunity to obtain improvements via a combination of ML and CO, multiple lines of research have emerged, such as Decision Focused Learning, Constrained Reinforcement Learning, or Algorithm Configuration. While existing methods have obtained a good measure of success, to the best of the authors knowledge no existing method can deal with all the challenges we have identified. Ideally, one wishes to obtain a solution policy capable of providing feasible (and high-quality) solutions, handling robustness, taking advantage of existing data, and with a reasonable computational load.

We consider the problem of simultaneous learning in stochastic games with many players in the finite-horizon setting. While the typical target solution for a stochastic game is a Nash equilibrium, this is intractable with many players. We instead focus on variants of {\it correlated equilibria}, such as those studied for extensive-form games. We begin with a hardness result for the adversarial MDP problem: even for a horizon of 3, obtaining sublinear regret against the best non-stationary policy is \textsf{NP}-hard when both rewards and transitions are adversarial. This implies that convergence to even the weakest natural solution concept -- normal-form coarse correlated equilbrium -- is not possible via black-box reduction to a no-regret algorithm even in stochastic games with constant horizon (unless $\textsf{NP}\subseteq\textsf{BPP}$). Instead, we turn to a different target: algorithms which {\it generate} an equilibrium when they are used by all players. Our main result is algorithm which generates an {\it extensive-form} correlated equilibrium, whose runtime is exponential in the horizon but polynomial in all other parameters. We give a similar algorithm which is polynomial in all parameters for "fast-mixing" stochastic games. We also show a method for efficiently reaching normal-form coarse correlated equilibria in "single-controller" stochastic games which follows the traditional no-regret approach. When shared randomness is available, the two generative algorithms can be extended to give simultaneous regret bounds and converge in the traditional sense.

Growing up in north London, the child of a Greek Cypriot father and a Chinese Singaporean mother, Hassabis was a child prodigy in chess from the age of 4. He began writing his own computer games at 8, created one of the first video games to use AI at 17, and founded his own video game company not long after graduating from Cambridge University at 20. So perhaps it makes sense that Hassabis's AI startup DeepMind, founded in 2010 and sold to Google just four years later, would achieve its first major successes with AI models that used deep reinforcement learning to rapidly master video games like Space Invaders and Q*bert without any knowledge of the actual rules. That was followed with AlphaGo, which learned the ancient strategy board game of Go and would in 2017 defeat the world's number one human player -- an event that did perhaps more than anything else to awaken the world to the rapid progress of AI. New models could dominate a variety of games even faster, reducing the time and human intervention needed to acquire mastery.

How would an artificial intelligence (AI) decide what to do? One common approach in AI research is called "reinforcement learning". Reinforcement learning gives the software a "reward" defined in some way, and lets the software figure out how to maximise the reward. This approach has produced some excellent results, such as building software agents that defeat humans at games like chess and Go, or creating new designs for nuclear fusion reactors. However, we might want to hold off on making reinforcement learning agents too flexible and effective.

How would an artificial intelligence (AI) decide what to do? One common approach in AI research is called "reinforcement learning". Reinforcement learning gives the software a "reward" defined in some way, and lets the software figure out how to maximise the reward. This approach has produced some excellent results, such as building software agents that defeat humans at games like chess and Go, or creating new designs for nuclear fusion reactors. However, we might want to hold off on making reinforcement learning agents too flexible and effective.

Growing up in north London, the child of a Greek Cypriot father and a Chinese Singaporean mother, Hassabis was a child prodigy in chess from the age of 4. He began writing his own computer games at 8, created one of the first video games to use AI at 17, and founded his own video game company not long after graduating from Cambridge University at 20. So perhaps it makes sense that Hassabis's AI startup DeepMind, founded in 2010 and sold to Google just four years later, would achieve its first major successes with AI models that used deep reinforcement learning to rapidly master video games like Space Invaders and Q*bert without any knowledge of the actual rules. That was followed with AlphaGo, which learned the ancient strategy board game of Go and would in 2017 defeat the world's number one human player -- an event that did perhaps more than anything else to awaken the world to the rapid progress of AI. New models could dominate a variety of games even faster, reducing the time and human intervention needed to acquire mastery.

In probably approximately correct (PAC) reinforcement learning (RL), an agent is required to identify an $\epsilon$-optimal policy with probability $1-\delta$. While minimax optimal algorithms exist for this problem, its instance-dependent complexity remains elusive in episodic Markov decision processes (MDPs). In this paper, we propose the first nearly matching (up to a horizon squared factor and logarithmic terms) upper and lower bounds on the sample complexity of PAC RL in deterministic episodic MDPs with finite state and action spaces. In particular, our bounds feature a new notion of sub-optimality gap for state-action pairs that we call the deterministic return gap. While our instance-dependent lower bound is written as a linear program, our algorithms are very simple and do not require solving such an optimization problem during learning. Their design and analyses employ novel ideas, including graph-theoretical concepts (minimum flows) and a new maximum-coverage exploration strategy.

Unmanned aerial vehicles (UAVs) are finding use in applications that place increasing emphasis on robustness to external disturbances including extreme wind. However, traditional multirotor UAV platforms do not directly sense wind; conventional flow sensors are too slow, insensitive, or bulky for widespread integration on UAVs. Instead, drones typically observe the effects of wind indirectly through accumulated errors in position or trajectory tracking. In this work, we integrate a novel flow sensor based on micro-electro-mechanical systems (MEMS) hot-wire technology developed in our prior work onto a multirotor UAV for wind estimation. These sensors are omnidirectional, lightweight, fast, and accurate. In order to achieve superior tracking performance in windy conditions, we train a `wind-aware' residual-based controller via reinforcement learning using simulated wind gusts and their aerodynamic effects on the drone. In extensive hardware experiments, we demonstrate the wind-aware controller outperforming two strong `wind-unaware' baseline controllers in challenging windy conditions. See: https://youtu.be/KWqkH9Z-338.

This paper studies the computational offloading of CNN inference in dynamic multi-access edge computing (MEC) networks. To address the uncertainties in communication time and Edge servers' available capacity, we use early-exit mechanism to terminate the computation earlier to meet the deadline of inference tasks. We design a reward function to trade off the communication, computation and inference accuracy, and formulate the offloading problem of CNN inference as a maximization problem with the goal of maximizing the average inference accuracy and throughput in long term. To solve the maximization problem, we propose a graph reinforcement learning-based early-exit mechanism (GRLE), which outperforms the state-of-the-art work, deep reinforcement learning-based online offloading (DROO) and its enhanced method, DROO with early-exit mechanism (DROOE), under different dynamic scenarios. The experimental results show that GRLE achieves the average accuracy up to 3.41x over graph reinforcement learning (GRL) and 1.45x over DROOE, which shows the advantages of GRLE for offloading decision-making in dynamic MEC.

The multi-vehicle pursuit (MVP), as a problem abstracted from various real-world scenarios, is becoming a hot research topic in Intelligent Transportation System (ITS). The combination of Artificial Intelligence (AI) and connected vehicles has greatly promoted the research development of MVP. However, existing works on MVP pay little attention to the importance of information exchange and cooperation among pursuing vehicles under the complex urban traffic environment. This paper proposed a graded-Q reinforcement learning with information-enhanced state encoder (GQRL-IESE) framework to address this hierarchical collaborative multi-vehicle pursuit (HCMVP) problem. In the GQRL-IESE, a cooperative graded Q scheme is proposed to facilitate the decision-making of pursuing vehicles to improve pursuing efficiency. Each pursuing vehicle further uses a deep Q network (DQN) to make decisions based on its encoded state. A coordinated Q optimizing network adjusts the individual decisions based on the current environment traffic information to obtain the global optimal action set. In addition, an information-enhanced state encoder is designed to extract critical information from multiple perspectives and uses the attention mechanism to assist each pursuing vehicle in effectively determining the target. Extensive experimental results based on SUMO indicate that the total timestep of the proposed GQRL-IESE is less than other methods on average by 47.64%, which demonstrates the excellent pursuing efficiency of the GQRL-IESE. Codes are outsourced in https://github.com/ANT-ITS/GQRL-IESE.