Floating-Point Formats and Deep Learning

Floating-point formats are not the most glamorous or (frankly) the important consideration when working with deep learning models: if your model isn't working well, then your floating-point format certainly isn't going to save you! However, past a certain point of model complexity/model size/training time, your choice of floating-point format can have a significant impact on your model training times and even performance. Here's how the rest of this post is structured: It's more likely than you think! It's been known for quite some time that deep neural networks can tolerate lower numerical precision. High-precision calculations turn out not to be that useful in training or inferencing neural networks: the additional precision confers no benefit while being slower and less memory-efficient.