Generalist Forecasting with Frozen Video Models via Latent Diffusion