Supplementary Materials for NAR-Former V2: Rethinking Transformer for Universal Neural Network Representation Learning

Right: Normalized attention scores processed by two different normalization methods. Table 1: Performance of searched architectures using different NAS algorithms in DARTS [ 7 ] space on CIFAR-10 [ 5 ]. The inference latency was measured on a machine with GeForce RTX 3090 GPU. The batch size was set to 1. Encode(ms) Infer(ms) Total(ms) NAR-Former 2.4784 17.4864 19.9648 NAR-Former V2 2.3722 5.2276 7.5998 may be somewhat different. Due to the softmax, Eq. ( 5) focuses almost all attention on the current The Eq. ( 2) restricts attention to connected nodes by introducing the adjacency matrix.