Time-Varying LoRA: Towards Effective Cross-Domain Fine-Tuning of Diffusion Models