GPU-Accelerated Atari Emulation for Reinforcement Learning

We designed and implemented a CUDA port of the Atari Learning Environment (ALE), a system for developing and evaluating deep reinforcement algorithms using Atari games. Our CUDA Learning Environment (CuLE) overcomes many limitations of existing CPUbased Atari emulators and scales naturally to multi-GPU systems. It leverages the parallelization capability of GPUs to run thousands of Atari games simultaneously; by rendering frames directly on the GPU, CuLE avoids the bottleneck arising from the limited CPU-GPU communication bandwidth. Figure 1: In a typical DRL system, environments run As a result, CuLE is able to generate between 40M on CPUs, whereas GPUs execute DNN operations. The and 190M frames per hour using a single GPU, a finding limited CPU-GPU communication bandwidth and small that could be previously achieved only through a cluster set of CPU environments prevent full GPU utilization. of CPUs. We demonstrate the advantages of CuLE by effectively training agents with traditional deep reinforcement learning algorithms and measuring the utilization benchmark for DRL [4, 14], and still represent a challenging and throughput of the GPU. Our analysis further highlights set for the development of new DRL methods.