Consequently, schemes for numerical discretization are a key element of scientific computing, and one of the main challenges is to determine a good trade-off between the required accuracy and numerical efficiency. While the standard approach to developing quadrature rules and numerical schemes to solve ODEs is based on Taylor series expansions and the associated error bounds determined by higher-order derivatives [2], the advances in data science and machine learning have recently fueled the development of alternative concepts that are based on training data. Most of these approaches are developed with the aim to efficiently compute the numerical solution of dynamical systems of high complexity, see, for instance, [18, 25, 15, 19, 20, 12], where the flow map F that takes a state x at time t to a future state x(t + t) is approximated. In contrast to that, our work addresses the task of efficiently performing numerical integration for integrands or differential equations of a given problem class to a desired accuracy. To this end, the next sample point at which the integrand or force term is evaluated is determined by finding an optimal trade-off between the two conflicting criteria accuracy and numerical efficiency. This task is carried out by a reinforcement learning algorithm which, taking past function evaluations and learned knowledge about the problem class into account, determines the next sample point at which to evaluate the function of interest. The efficiency of the proposed approach in comparison to state-of-the-art methods will be demonstrated using examples from the area of computing integrals (quadrature) as well as numerically solving differential equations.

This thesis rigorously studies fundamental reinforcement learning (RL) methods in modern practical considerations, including robust RL, distributional RL, and offline RL with neural function approximation. The thesis first prepares the readers with an overall overview of RL and key technical background in statistics and optimization. In each of the settings, the thesis motivates the problems to be studied, reviews the current literature, provides computationally efficient algorithms with provable efficiency guarantees, and concludes with future research directions. The thesis makes fundamental contributions to the three settings above, both algorithmically, theoretically, and empirically, while staying relevant to practical considerations.

Deep reinforcement learning is one of the most interesting branches ofartificial intelligence. It is behind some of the most remarkable achievements of the AI community, including beating human champions at board and video games, self-driving cars, robotics, and AI hardware design. Deep reinforcement learning leverages the learning capacity of deep neural networks to tackle problems that were too complex for classic RL techniques. Deep reinforcement learning is much more complicated than the other branches of machine learning. But in this post, I'll try to demystify it without going into the technical details.

The traveling salesman problem is a fundamental combinatorial optimization problem with strong exact algorithms. However, as problems scale up, these exact algorithms fail to provide a solution in a reasonable time. To resolve this, current works look at utilizing deep learning to construct reasonable solutions. Such efforts have been very successful, but tend to be slow and compute intensive. This paper exemplifies the integration of entropic regularized optimal transport techniques as a layer in a deep reinforcement learning network. We show that we can construct a model capable of learning without supervision and inferences significantly faster than current autoregressive approaches. We also empirically evaluate the benefits of including optimal transport algorithms within deep learning models to enforce assignment constraints during end-to-end training.

Rich S. Sutton, a research scientist at DeepMind and computing science professor at the University of Alberta, explains the underlying formal problem like the Markov decision processes, core solution methods, dynamic programming, Monte Carlo methods, and temporal-difference learning in this in-depth tutorial.

In reinforcement learning, state representations are used to tractably deal with large problem spaces. State representations serve both to approximate the value function with few parameters, but also to generalize to newly encountered states. Their features may be learned implicitly (as part of a neural network) or explicitly (for example, the successor representation of \citet{dayan1993improving}). While the approximation properties of representations are reasonably well-understood, a precise characterization of how and when these representations generalize is lacking. In this work, we address this gap and provide an informative bound on the generalization error arising from a specific state representation. This bound is based on the notion of effective dimension which measures the degree to which knowing the value at one state informs the value at other states. Our bound applies to any state representation and quantifies the natural tension between representations that generalize well and those that approximate well. We complement our theoretical results with an empirical survey of classic representation learning methods from the literature and results on the Arcade Learning Environment, and find that the generalization behaviour of learned representations is well-explained by their effective dimension.

Welcome to our February 2022 monthly digest, where you can catch up with any AIhub stories you may have missed, get the low-down on recent events, and much more. This month, we cover our latest New voices in AI interview, hear from a NeurIPS award winner, and get stuck into AAAI 2022. You may have seen the launch of our new series last month. In the latest episode, Isabel Cachola talks about how she got into AI and her work on interpretability of NLP models. In this interview, Bernard Koch tells us about research that won him, and co-authors Emily Denton, Alex Hanna and Jacob Foster, a best paper prize at NeurIPS 2021.

Magenta was started by the Google Brain team, which developed new deep learning and reinforcement learning algorithms for generating songs, …

Scientists at EPFL's Swiss Plasma Center and DeepMind have jointly developed a new method for controlling plasma configurations for use in nuclear fusion research. EPFL's Swiss Plasma Center (SPC) has decades of experience in plasma physics and plasma control methods. DeepMind is a scientific discovery company acquired by Google in 2014 that's committed to'solving intelligence to advance science and humanity. Together, they have developed a new magnetic control method for plasmas based on deep reinforcement learning, and applied it to a real-world plasma for the first time in the SPC's tokamak research facility, TCV. Their study has just been published in Nature.

We study the convergence of the actor-critic algorithm with nonlinear function approximation under a nonconvex-nonconcave primal-dual formulation. Stochastic gradient descent ascent is applied with an adaptive proximal term for robust learning rates. We show the first efficient convergence result with primal-dual actor-critic with a convergence rate of $\mathcal{O}\left(\sqrt{\frac{\ln \left(N d G^2 \right)}{N}}\right)$ under Markovian sampling, where $G$ is the element-wise maximum of the gradient, $N$ is the number of iterations, and $d$ is the dimension of the gradient. Our result is presented with only the Polyak-\L{}ojasiewicz condition for the dual variables, which is easy to verify and applicable to a wide range of reinforcement learning (RL) scenarios. The algorithm and analysis are general enough to be applied to other RL settings, like multi-agent RL. Empirical results on OpenAI Gym continuous control tasks corroborate our theoretical findings.