Accelerating Deep Neural Networks – Towards Data Science – Medium

Neural networks are "slow" for many reasons, including load/store latency, shuffling data in and out of the GPU pipeline, the limited width of the pipeline in the GPU (as mapped by the compiler), the unnecessary extra precision in most neural network calculations (lots of tiny numbers that make no difference to the outcome), the sparsity of input data (lots of 0s), and many many other factors. How can we make deep neural network training, testing, and predictions faster? One way is to write faster algorithms, like the relu activation function, which is much faster than tanh and sigmoid, and another is to write better compilers to map the neural network into the hardware. A third approach is what I want to tell you about today. Making better hardware, and by better I mean faster processing speed.