IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures.69

CuLE is available at https://github.com/NVlabs/cule.

We introduce CuLE (CUDA Learning Environment), a CUDA port of the Atari Learning Environment (ALE) which is used for the development of deep reinforcement algorithms. CuLE overcomes many limitations of existing CPU-based emulators and scales naturally to multiple GPUs. It leverages GPU parallelization to run thousands of games simultaneously and it renders frames directly on the GPU, to avoid the bottleneck arising from the limited CPU-GPU communication bandwidth. CuLE generates up to 155M frames per hour on a single GPU, a finding previously achieved only through a cluster of CPUs. Beyond highlighting the differences between CPU and GPU emulators in the context of reinforcement learning, we show how to leverage the high throughput of CuLE by effective batching of the training data, and show accelerated convergence for A2C+V-trace. CuLE is available at https://github.com/NVlabs/cule.

Weaknesses: My main concern is that results seem to be contradictory to what the authors claimed as the benefit of leveraging GPU accelerations. Specifically, in the "impact statement" the authors described CuLE can "provide access to an accelerated training environment to researchers with limited computational capabilities," but the results show the acceleration won't take into effect unless you use more computation---Figure 2, CuLE runs slower than OpenAI when using a fewer number of environments. If someone can only afford to run 100 environments, would this mean CuLE is not useful here? The limitation of the memory has been noted in the paper which is good. I was confused when looking at Table 3. First, why is there no 120 envs experiment for CuLE?

We introduce CuLE (CUDA Learning Environment), a CUDA port of the Atari Learning Environment (ALE) which is used for the development of deep reinforcement algorithms. CuLE overcomes many limitations of existing CPU-based emulators and scales naturally to multiple GPUs. It leverages GPU parallelization to run thousands of games simultaneously and it renders frames directly on the GPU, to avoid the bottleneck arising from the limited CPU-GPU communication bandwidth. CuLE generates up to 155M frames per hour on a single GPU, a finding previously achieved only through a cluster of CPUs. Beyond highlighting the differences between CPU and GPU emulators in the context of reinforcement learning, we show how to leverage the high throughput of CuLE by effective batching of the training data, and show accelerated convergence for A2C V-trace.

We designed and implemented a CUDA port of the Atari Learning Environment (ALE), a system for developing and evaluating deep reinforcement algorithms using Atari games. Our CUDA Learning Environment (CuLE) overcomes many limitations of existing CPUbased Atari emulators and scales naturally to multi-GPU systems. It leverages the parallelization capability of GPUs to run thousands of Atari games simultaneously; by rendering frames directly on the GPU, CuLE avoids the bottleneck arising from the limited CPU-GPU communication bandwidth. Figure 1: In a typical DRL system, environments run As a result, CuLE is able to generate between 40M on CPUs, whereas GPUs execute DNN operations. The and 190M frames per hour using a single GPU, a finding limited CPU-GPU communication bandwidth and small that could be previously achieved only through a cluster set of CPU environments prevent full GPU utilization. of CPUs. We demonstrate the advantages of CuLE by effectively training agents with traditional deep reinforcement learning algorithms and measuring the utilization benchmark for DRL [4, 14], and still represent a challenging and throughput of the GPU. Our analysis further highlights set for the development of new DRL methods.