Gradient Editing On The Fly in Deep Neural Networks

The gradient is an essential ingredient in the world of deep learning. It powers one of the most popular algorithms -- gradient descent, which optimizes the parameters of neural networks by repeatedly changing their value based on the current gradient of the model's cost with respect to these parameters. Although being a simple approach in each update, the gradient descent family of algorithms, in particular stochastic gradient descent, has been one of the driving forces on the good generalization performance of deep learning models. However, much focus has been given to the algorithm itself instead of the gradient. Part of the reason is that the gradient is mostly automatically calculated and plugged into the optimization procedure.