Improving Fine-Grained Control via Aggregation of Multiple Diffusion Models