How Floating Point Numbers Work

Normally, python and numerical computing libraries like numpy or PyTorch handle this behind the scenes. But understanding the details can help you avoid otherwise unexpected errors and speed up many machine learning computations. For example, Google's Tensor Processing Units (TPUs) use a modified floating point format to substantially improve computational efficiency while trying to maintain good results. In this article we'll dig into the nuts and bolts of floating point numbers, cover the edge cases (numerical underflow and overflow), and close with applications: TPU's bfloat16 format and HDR imaging. The main background assumed is that you understand how to count in binary, as well as how binary fractions work. Let's briefly review counting to 5 in binary: 0, 1, 10, 11, 100, 101.