Despite a series of recent successes in reinforcement learning (RL), many RL algorithms remain sensitive to hyperparameters. As such, there has recently been interest in the field of AutoRL, which seeks to automate design decisions to create more general algorithms. Recent work suggests that population based approaches may be effective AutoRL algorithms, by learning hyperparameter schedules on the fly. In particular, the PB2 algorithm is able to achieve strong performance in RL tasks by formulating online hyperparameter optimization as time varying GP-bandit problem, while also providing theoretical guarantees. However, PB2 is only designed to work for \emph{continuous} hyperparameters, which severely limits its utility in practice. In this paper we introduce a new (provably) efficient hierarchical approach for optimizing \emph{both continuous and categorical} variables, using a new time-varying bandit algorithm specifically designed for the population based training regime. We evaluate our approach on the challenging Procgen benchmark, where we show that explicitly modelling dependence between data augmentation and other hyperparameters improves generalization.

People Can Fly is one of the leading independent AAA games development studios with an international team of hundreds of talented individuals working from offices located in Poland, UK, US, and Canada, and from all over the world thanks to our remote work programs. Founded in 2002, we made our mark on the shooter genre with titles such as Painkiller, Bulletstorm, Gears of War: Judgment, and Outriders. We are one of the most experienced Unreal Engine studios in the industry and we are expanding it with in-house solutions called PCF Framework. Our creative teams are currently working on several exciting titles: Gemini is our new project being developed with Square Enix; Bifrost, Victoria and Dagger are projects we're growing in the self-publishing model. We also have one project in the concept phase – Red; as well as two projects in VR technology – Green Hell VR and Thunder - a new project based on one of the IPs from the Group's portfolio.

People Can Fly is one of the leading independent AAA games development studios with an international team of hundreds of talented individuals working from offices located in Poland, UK, US, and Canada, and from all over the world thanks to our remote work programs. Founded in 2002, we made our mark on the shooter genre with titles such as Painkiller, Bulletstorm, Gears of War: Judgment, and Outriders. We are one of the most experienced Unreal Engine studios in the industry and we are expanding it with in-house solutions called PCF Framework. Our creative teams are currently working on several exciting titles: Gemini is our new project being developed with Square Enix; Bifrost, Victoria and Dagger are projects we're growing in the self-publishing model. We also have one project in the concept phase – Red; as well as two projects in VR technology – Green Hell VR and Thunder - a new project based on one of the IPs from the Group's portfolio.

Planning--the ability to synthesize a course of action to achieve desired goals--is an important part of intelligent agency and has thus received significant attention within AI for more than 30 years. Work on efficient planning algorithms still continues to be a hot topic for research in AI and has led to several exciting developments in the past few years. This article provides a tutorial introduction to all the algorithms and approaches to the planning problem in AI. To fulfill this ambitious objective, I introduce a generalized approach to plan synthesis called refinement planning and show that in its various guises, refinement planning subsumes most of the algorithms that have been, or are being, developed. It is hoped that this unifying overview provides the reader with a brand-name-free appreciation of the essential issues in planning.