Our method also has no trainable parameter. The method is applied directly in the feature space at test time and does not intervene in training process.

Adam has been shown to outperform gradient descent on large language models by a larger margin than on other tasks, but it is unclear why. We show that a key factor in this performance gap is the heavy-tailed class imbalance found in language tasks.

Despite the widespread success of Transformers across various domains, their optimization guarantees in large-scale model settings are not well-understood.