Review -- Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?

The interaction with all the other white tokens can be achieved when sMLP is executed twice. It consists of three branches: two of them are responsible for mixing information along horizontal and vertical directions respectively and the other path is the identity mapping. The output of the three branches are concatenated and processed by a pointwise convolution to obtain the final output. We can see that MLP-Mixer cannot afford a high-resolution input or the pyramid processing, as the computational complexity grows with N². In contrast, the computational complexity of the proposed sMLP grows with N N.