Parameter-Efficient Generative Modeling with Controlled Vector Fields