Pre-training Differentially Private Models with Limited Public Data

The superior performance of large foundation models can be attributed to the use of massive amounts of high-quality data.