An In-depth Study of Stochastic Backpropagation

In particular, we discuss the following: Section 8.1 derives the gradient calculation for attention layers. Section 8.4 investigates the insights on the gradient keep-ratios and gradient keep masks on Section 8.6 compares the model similarity between with and without applying SBP . In section 3.2, we provide the gradient calculation of linear layers (or PW-Conv) and general convolutional layers for the backward phase of SBP . Eq. (18) and it will be an approximated version of its original case as well. The MLP sub-block is equivalent to two PW-Conv or linear layers.