State Fourier Diffusion Language Model (SFDLM): A Scalable, Novel Iterative Approach to Language Modeling

School of Infomation University of California Berkeley, CA 94720-1776, USA Andreas Lemos School of Infomation University of California Berkeley, CA 94720-1776, USA Editor: Abstract In recent years, diffusion-based methods have emerged as a powerful paradigm for generative modeling, initially introduced by Sohl-Dickstein et al. (2015) for continuous data and subsequently refined by Ho et al. (2020) in the context of image generation. Although discrete diffusion for natural language processing has been explored to a lesser extent, it shows promise for tasks requiring iterative denoising of token-based data. In standard approaches to text generation, transformers dominate, but their reliance on self-attention often incurs high computational costs. This paper introduces a fully diffusion-driven discrete text generation model built without any transformer or large convolution modules. Instead, the model integrates structured state-space dynamics in the time domain with a novel Complex Fourier Multi-Layer Perceptron (MLP) module that operates in the frequency domain. The forward noising process randomly samples the vocabulary to replace tokens with a controlled probability, while the learned reverse model systematically reverts corrupted sequences toward their original states. Experiments on text datasets demonstrate the model's capacity to iteratively refine noised sequences and produce coherent token predictions with the Complex Fourier MLP, leading to enhanced flexibility in shifting both amplitude and phase.