Running out of GPU memory has become a main bottleneck for large-scale DNN training. How to reduce the memory footprint during training has received intensive research attention. We find that previous gradient accumulation reduces activation memory but fails to be compatible with gradient memory reduction due to a contradiction between preserving gradients and releasing gradients. To address this issue, we propose a novel optimizer accumulation method for Adam, named Adam Accumulation (AdamA), which enables reducing both activation and gradient memory. Specifically, AdamA directly integrates gradients into optimizer states and accumulates optimizer states over micro-batches, so that gradients can be released immediately after use. We mathematically and experimentally demonstrate AdamA yields the same convergence properties as Adam. Evaluated on transformer-based models, AdamA achieves up to 23% memory reduction compared to gradient accumulation with less than 2% degradation in training throughput. Notably, AdamA can work together with memory reduction methods for optimizer states to fit 1.26x~3.14x larger models over PyTorch and DeepSpeed baseline on GPUs with different memory capacities.

For die-hard fans of the Syfy drama "Battlestar Galactica," the closing night reunion panel at the ATX Television Festival was a treat-filled trip down memory lane. Seven key cast members -- Edward James Olmos (Admiral Adama), Mary McDonnell (President Laura Roslin), Katee Sackhoff (Starbuck), James Callis (Gaius Baltar), Tricia Helfer (Number Six), Grace Park (Boomer/Number Eight) and Michael Trucco (Sam Anders) -- and executive producer Ronald D. Moore came together to celebrate the groundbreaking series that was as much post-9/11 allegory as it was grand space adventure. The "BSG" gang still clearly enjoy each other's company and had both the audience and each other cracking up throughout the nearly two-hour Q&A session at Austin's Paramount Theatre. From awkward sex scenes (Callis and Helfer) to uncontrollable giggles (McDonnell and Sackhoff) to Olmos's Adama-like leadership qualities, the cast mates and Moore fondly recalled funny and touching moments from their time on the series, which ran from 2004 to 2009. Callis recalled a scene in which he fell and hit his head requiring a trip to an emergency room.