Review for NeurIPS paper: Ultra-Low Precision 4-bit Training of Deep Neural Networks

Weaknesses: Ablation study Figure 6a: 1x1 convs often represent a large portion of the FLOPs in networks, and for that reason works like ShuffleNet break them into group convolutions. Is there anything fundamental about the 1x1 conv to justify putting it in FP8? What is the effect of putting just the 3x3 layers in FP8? Especially if the 1x1 convs include conv in the identity branch, most of the conv layers would now be in FP8. Perhaps it should be stated what percentage of gradients are in 8-bit, although this may be captured in the estimated performance data. Edit: After the author feedback, I still believe that it is misleading to label the method as fully 4-bit when a significant number of layers are cast in FP8.