Reviews: Memory Efficient Adaptive Optimization

As larger-and-larger models are being trained on increasing amounts of data, on-device memory becomes a bottleneck. Efforts which reduce storage of auxiliary information including gradient moments and activations, allow for larger models and/or batches on same hardware. I endorse the paper for acceptance to NeurIPS. The paper is well-written and the results are impressive. I especially appreciate that the authors tested the algorithm on cutting-edge models such as ImageNet, BERT and Transformer MT, and also had an ablation on hyperparameters such as batch sizes (Figure 3b).