Real-time Inference and Extrapolation via a Diffusion-inspired Temporal Transformer Operator (DiTTO)