training

RTFormer is consist of several convolution blocks and RTFormerblocks,andRTFormerblockcontains differenttypes of attention. Table 2 shows the performance of RTFormer on ImageNet classification. The first three results of multi-head external attention are with r = [0.125,0.25,1]respectively. As illustrated in Table 3, we can find that multi-head self-attention achieves32.7 mIoU, which performs better than multi-head external attentions with different settings ofr. Multi-head external attention can achieve a good inference speed, which is benefit from its linear complexity and the design of sharing external parameter for multiple heads. However,theperformance ofmulti-headexternal attention is suboptimal, as the network capacity is limited by those designs.