DivControl: Knowledge Diversion for Controllable Image Generation