A Statistical Framework for Low-bitwidth Training of Deep Neural Networks

For training ResNet-50 on ImageNet, our 5-bit block Householder quantizer achieves only 0.5% validation accuracy loss relative to QA T, comparable to the existing INT8 baseline.