Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings

In recent years, diffusion models [1, 2], particularly latent diffusion models [3], have spearheaded high quality textto-image generation, and have been widely adopted by researchers and the general public alike. Trained on massive datasets like LAION-5B [4], these models have developed a broad understanding of visual concepts, enabling new creative and practical applications. Notably, tools like Stable Diffusion [3, 5] have been made readily accessible for general use. Building on this foundation, efficient adaptation methods such as parameter efficient fine-tuning (PEFT) [6, 7, 8], guidance based approaches [9, 10, 11], and pseudo-word generation [12] enable users to leverage this extensive pretraining for customizing models that can specialize on downstream tasks with smaller datasets. However, the rapid adoption of diffusion models has also raised significant privacy, ethical and legal concerns. One critical issue is the vulnerability of these models to privacy attacks, from membership inference [13], where an attacker determines whether a specific data point was used to train a particular model, to data extraction [14], which enables an attacker to reconstruct particular images from the training dataset. This issue is even more severe during the fine-tuning phase where the model is fine-tuned on smaller specialized datasets from a possibly different domain and each data record has more impact on the final model. This risk underscores the importance of privacy-preserving technologies, particularly as diffusion models often rely on vast datasets scraped from the internet without explicit consent from content owners.