We provide our source codes to ensure the reproducibility of our experimental results. Below we summarize several critical aspects w.r .tthe The datasets we used are all publicly accessible, which is introduced in Appendix E.1. For long-tailed subsets, we strictly follows previous work [29] on CIFAR-100-L T to avoid the bias attribute to the sampling randomness. On ImageNet-L T and Places-L T, we employ the widely-used data split first introduced in [44]. All the experiments are conducted on NVIDIA GeForce RTX 3090 with Python 3.7 and Pytorch 1.7.

Despite the widespread success of Transformers across various domains, their optimization guarantees in large-scale model settings are not well-understood.

In this section, we provide two different ways to prove Theorem 2. The first one is more straightforward and directly differentiates through the term To solve this issue, we introduce the following variational representation: Lemma 1. With Jensen's inequality, we have: log null null As introduced in Equation (9) in Section 2.3, the divergence corresponding to the This is a direct consequence of Lemma 2. It can also be verified by checking the PS-CD Lemma 3. When 1 γ < 0, we have: S We first make the following assumption, which is similar to the one used in [4, 47]: Assumption 1. The assumption is typically easy to enforce in practice. In this section, we analyze the convergence property of the PS-CD algorithm presented in Algorithm 1. We have the following theorem that characterizes the convergence property of Algorithm 2: Theorem 5. Monte Carlo estimation will incur additional approximation error.