In this paper, we propose a number of novel techniques and numerical representation formats that enable, for the very first time, the precision of training systems to be aggressively scaled from 8-bits to 4-bits. To enable this advance, we explore a novel adaptive Gradient Scaling technique (Gradscale) that addresses the challenges of insufficient range and resolution in quantized gradients as well as explores the impact of quantization errors observed during model training. We theoretically analyze the role of bias in gradient quantization and propose solutions that mitigate the impact of this bias on model convergence. Finally, we examine our techniques on a spectrum of deep learning models in computer vision, speech, and NLP. In combination with previously proposed solutions for 4-bit quantization of weight and activation tensors, 4-bit training shows a non-significant loss in accuracy across application domains while enabling significant hardware acceleration (> 7X over state-of-the-art FP16 systems).

Weaknesses: Ablation study Figure 6a: 1x1 convs often represent a large portion of the FLOPs in networks, and for that reason works like ShuffleNet break them into group convolutions. Is there anything fundamental about the 1x1 conv to justify putting it in FP8? What is the effect of putting just the 3x3 layers in FP8? Especially if the 1x1 convs include conv in the identity branch, most of the conv layers would now be in FP8. Perhaps it should be stated what percentage of gradients are in 8-bit, although this may be captured in the estimated performance data. Edit: After the author feedback, I still believe that it is misleading to label the method as fully 4-bit when a significant number of layers are cast in FP8.

Fast training and model compression are important issues when applying machine learning techniques in practice. The proposed 4-bit training method in this paper is novel. The empirical experiments are comprehensive and the results are promising. A minor issue is that it does not seem very clear how hardware could well support this method. Please add some discussions on this in the final version.

In this paper, we propose a number of novel techniques and numerical representation formats that enable, for the very first time, the precision of training systems to be aggressively scaled from 8-bits to 4-bits. To enable this advance, we explore a novel adaptive Gradient Scaling technique (Gradscale) that addresses the challenges of insufficient range and resolution in quantized gradients as well as explores the impact of quantization errors observed during model training. We theoretically analyze the role of bias in gradient quantization and propose solutions that mitigate the impact of this bias on model convergence. Finally, we examine our techniques on a spectrum of deep learning models in computer vision, speech, and NLP. In combination with previously proposed solutions for 4-bit quantization of weight and activation tensors, 4-bit training shows a non-significant loss in accuracy across application domains while enabling significant hardware acceleration ( 7X over state-of-the-art FP16 systems).