The Fallacy of the FLOPS - Neural Magic

Everything we know about memory requirements in machine learning may be wrong. Today, when data scientists process deep learning models using a "throughput computing" device like a GPU, TPU, or similar hardware accelerator, they're likely faced with a decision to shrink their model or input size to fit within the device's memory limitations. Training a large, deep neural network (or even a wide, shallow one) on a single GPU, in many cases, may be impossible. Ever wonder why on the original Resnet 152, the winner of the ILSVRC-2015 image detection competition had 152 layers and not 153? Is it a coincidence that the parameters in 152 layers have a memory footprint of slightly less than 12G, while 153 layers go beyond 12G (the standard size of GPU memory at the time)?