CUDA Neural Networks

CUDA stands for Compute Unified Device Architecture, and it's the reason popular deep learning libraries like Tensorflow & PyTorch are considered "GPU-accelerated". CUDA is Nvidia's programming platform that enables developers to leverage the full parallel processng capabilities of GPUs for deep learning applications. Almost all of the major deep learning libraries use CUDA under the hood, but it's not really something that most developers think about often. In this episode, I'll demo some progressively more complex CUDA examples by Nvidia to show you how using CUDA results in algorithmic speedups. We'll use Nvidia's profiler to clock speeds, then we'll analyze a pure-CUDA neural network by Sergey Bugrov to understand how a full neural pipeline on the GPU looks like.