State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory

State-space models have gained popularity in sequence modelling due to their simple and efficient network structures. However, the absence of nonlinear activation along the temporal direction limits the model's capacity.