Augmented Shortcuts for Vision Transformers (Supplementary Material) Y ehui T ang

Following that original shortcut connections exist in both MSA and MLP modules, the proposed augmented shortcuts are also embedded into the MLP module (Eq. 10 in the main paper). S.4), the diversity in a block of Aug-ViT model is: Deep neural networks with trainable activations and controlled lipschitz constant.