Ultra-LowPrecision4-bitTrainingofDeepNeural Networks

While 8-bit floating point formats appeartobesufficient fortraining [14,19],4-bit gradient representations appear challenging from a quantization error (rounding), precision and dynamic range perspective.