By replacing the linear operation with our approximated one in transformers, we can achieve up to 2.7 peak memory reduction with almost no accuracy drop and enables up to6.4 larger batch size.

Although these methods are "parameter-efficient", they actually cannot reduce the The solution to the above convex problem is the distribution defined in Equation (3). Var[f (j) |j D\C ]. (16) By combining the above two inequality, we have Algorithm 2. For the ease of illustration, we ignore the sequential length. Cache is used for saving the norm of output gradient Z. end procedure F E.2 More Experimental Speed Analysis In Table 3, "Fwd", "Bwd", and "F-B" are the time of forward pass, the time of backward Latency (ms) of Forward and Backward pass. We give the detailed hyper-parameter setting in this section. The computational infrastructure information is given in Table 4.

With the rapid growth in model size, fine-tuning the large pre-trained language model has become increasingly difficult due to its extensive memory usage. Previous works usually focus on reducing the number of trainable parameters in the network. While the model parameters do contribute to memory usage, the primary memory bottleneck during training arises from storing feature maps, also known as activations, as they are crucial for gradient calculation. Notably, neural networks are usually trained using stochastic gradient descent. We argue that in stochastic optimization, models can handle noisy gradients as long as the gradient estimator is unbiased with reasonable variance. Following this motivation, we propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance, which only requires storing the sub-sampled activations for calculating the gradient. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones. By replacing the linear operation with our approximated one in transformers, we can achieve up to 2.7$\times$ peak memory reduction with almost no accuracy drop and enables up to $6.4\times$ larger batch size. Under the same hardware, WTA-CRS enables better down-streaming task performance by applying larger models and/or faster training speed with larger batch sizes.