Fourier with Deep Learning in Sequence Translation

As deep learning architectures are a technique to write a learning system where gradients are the only necessary requirements. FNet uses the Fourier transform to replace the Self-Attention of BERT [3]. The Fourier transform is a technique to embedding an existing function by one using the sinusoidal functions as a basis which originally was though to take O(n²) time complexity where n exists as the size of the input. The Cooley-Tukey Paper from Scripps described a method which takes O(n log n) in 1965 [1]. The Fast Fourier Transform was found because of performing the calculations by hand, a possible reason why people use pen and paper.