CS231n Convolutional Neural Networks for Visual Recognition

In the previous sections we've discussed the static parts of a Neural Networks: how we can set up the network connectivity, the data, and the loss function. This section is devoted to the dynamics, or in other words, the process of learning the parameters and finding good hyperparameters. In theory, performing a gradient check is as simple as comparing the analytic gradient to the numerical gradient. In practice, the process is much more involved and error prone. This requires you to evaluate the loss function twice to check every single dimension of the gradient (so it is about 2 times as expensive), but the gradient approximation turns out to be much more precise. To see this, you can use Taylor expansion of \(f(x h)\) and \(f(x-h)\) and verify that the first formula has an error on order of \(O(h)\), while the second formula only has error terms on order of \(O(h 2)\) (i.e. it is a second order approximation). What are the details of comparing the numerical gradient \(f'_n\) and analytic gradient \(f'_a\)? That is, how do we know if the two are not compatible? You might be temped to keep track of the difference \(\mid f'_a - f'_n \mid \) or its square and define the gradient check as failed if that difference is above a threshold.