Rethinking cluster-conditioned diffusion models