Efficient Time Series Processing for Transformers and State-Space Models through Token Merging