How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?