How to Increase Training Performance Through Memory Optimization

One of the keys to optimizing the runtime performance of your deep neural network (DNN) training workloads is to maximize the utilization of your training instance's resources. This is particularly true of the resources of the GPU, or other training accelerator, typically the most expensive component of your training device. Our focus in this post will be on the memory utilization of the GPU (or alternative training accelerator). For additional tips on training performance optimization be sure to check out some of our other blog posts (e.g. For the sake of simplicity, whenever we refer to GPU memory, we are referring more generally to the memory of any training accelerator, including GPU, Google Cloud TPU, Habana Gaudi, etc. The most basic example of GPU memory optimization is increasing your batch size to increase the memory utilization up to as close to 100% as possible. Generally speaking (but not always), your overall training throughput will increase.