Lessons for Improving Training Performance -- Part 2

Over the past nine months, the input pipeline part of deep learning training jobs in TensorFlow has become significantly more efficient. In this post, we investigate the performance impact of those TensorFlow changes and discuss futures on the horizon that will continue to impact full-stack performance. In Part 1 of this blog, we discussed the performance benefits of switching to lower precision and higher batch size during training. Both precision and batch size have "optimal" values for various workloads which have evolved over time. Beyond those major, well-known parameters, the entire software stack is evolving to improve training job throughput.