One of the grand challenges of reinforcement learning is the ability to generalize to new tasks. However, general agents require a set of rich, diverse tasks to train on. Designing a'foundation environment' for such tasks is tricky - the ideal environment would support a range of emergent phenomena, an expressive task space, and fast runtime. To take a step towards addressing this research bottleneck, this work presents Powderworld, a lightweight yet expressive simulation environment running directly on the GPU. Within Powderworld, two motivating challenges distributions are presented, one for world-modelling and one for reinforcement learning. Each contains hand-designed test tasks to examine generalization. Experiments indicate that increasing the environment's complexity improves generalization for world models and certain reinforcement learning agents, yet may inhibit learning in high-variance environments. Powderworld aims to support the study of generalization by providing a source of diverse tasks arising from the same core rules. One of the grand challenges of reinforcement learning (RL), and of decision-making in general, is the ability to generalize to new tasks. RL agents have shown incredible performance on single task settings (Berner et al., 2019; Lillicrap et al., 2015; Mnih et al., 2013), yet frequently stumble when presented with unseen challenges. Single-task RL agents are largely overfit on the tasks they are trained on (Kirk et al., 2021), limiting their practical use. In contrast, a general agent, which can robustly perform well on a wide range of novel tasks, can then be adapted to solve downstream tasks and unseen challenges. General agents greatly depend on a diverse set of tasks to train on. Recent progress in deep learning has shown that as the amount of data increases, so do generalization capabilities of trained models (Brown et al., 2020; Ramesh et al., 2021; Bommasani et al., 2021; Radford et al., 2021). Agents trained on environments with domain randomization or procedural generation capabilities transfer better to unseen test tasks Cobbe et al. (2020); Tobin et al. (2017); Risi & Togelius (2020); Khalifa et al. (2020).

Despite recent advances in RL research, the ability to generalize to new tasks remains one of the major issues in both reinforcement learning (RL) and decision-making. RL agents perform remarkably in a single-task setting but frequently make mistakes when faced with unforeseen obstacles. Additionally, single-task RL agents can largely overfit the tasks they are trained on, rendering them unsuitable for real-world applications. This is where a general agent that can successfully handle various unprecedented tasks and unforeseen difficulties can be useful. The vast majority of general agents are trained using a variety of diverse tasks.