The Journey, Not the Destination: How Data Guides Diffusion Models