End-To-End Latent Variational Diffusion Models for Inverse Problems in High Energy Physics