Using Galaxy Evolution as Source of Physics-Based Ground Truth for Generative Models