8-bit Numerical Formats for Deep Neural Networks

As already mentioned in Section 3.3, the same effect as changing the exponent bias for 8-bit representations of the gradients can be obtained by loss scaling. Instead of choosing a large exponent bias to prevent underflow, this method consists in scaling up the loss after the forward pass and scaling down the learning rate by the same factor before updating the weights or otherwise absorbing the scaling factor in the optimizer, as discussed in Section 2.5. We have tested algorithms that automate the selection of the loss scaling factor, to overcome the necessity to manually tune the bias for a low precision representation of the gradients. An adaptive loss scaling factor also allows one to react to changes in the long term statistics of the magnitudes of gradient components during training. Our experiments with different biases for quantization of the gradients suggest that the optimal range of representable values covers, or slightly clips, the largest occurring values and extends as far as possible to small values (see Appendix D, Figure 24).