Google's FNet: Mixing Tokens with Fourier Transforms

By replacing the attention sublayer with linear transformations, we are able to reduce the complexity and memory footprint of the Transformer architecture. We show that FNet offers an excellent compromise between speed, memory footprint, and accuracy, achieving 92% of the accuracy of BERT in a common classification transfer learning setup on the GLUE benchmark (Wang et al., 2018), but training seven times as fast on GPUs and twice as fast on TPUs Recent ML papers have been targeted at fiddling with transformer layers. It is quite interesting to see what works and what doesn't (even though we probably only see what works from those papers). Due to the significant usage of transformers, I think the last 6–12 months have been about optimizing them. This paper review is going to talk about changing the layers to improve the training speed, and the most interesting part is that it was done using Fourier transforms. Fourier Transform is a mathematical concept that can decompose a signal into its constituent frequencies.