Coding Neural Network - Gradient Checking · Imad Dabbura

In the previous post, Coding Neural Network - Forward Propagation and Backpropagation, we implemented both forward propagation and backpropagation in numpy. However, implementing backpropagation from scratch is usually more prune to bugs/errors. Therefore, it's necessary before running the neural network on training data to check if our implementation of backpropagation is correct. Before we start, let's revisit what back-propagation is: We loop over the nodes in reverse topological order starting at the final node to compute the derivative of the cost with respect to each edge's node tail. In other words, we compute the derivative of cost function with respect to all parameters, i.e $\frac{\partial J}{\partial \theta}$ where $\theta$ represents the parameters of the model.