Two Steps Forward, Two Steps Back

In this post, I'll start with a high-level review of what we've learned so far with neural networks and how they work up through a complete forward pass, and then conceptually walk through the back propagation technique to use gradient descent and adjust the randomized weight and bias values to align predictions more closely to actual labels. We will uncover some really neat math effects of using the ReLU activation function, and find out how the chain rule is applied to make finding the gradients across all of the layers a relatively low-effort process from a compute standpoint. This one is a little heavier than my normal post, by necessity, so brace yourselves! That said, hopefully this will help you understand how one of the more complicated concepts in machine learning today works in real-world applications. It starts with the Perceptron, which was the first machine learning algorithm developed and can be used to create a function that separated two classes, assuming a hyperplane exists that can do so.