Work was done when interned at Zhipu AI.

This paper contains uncensored toxic content that might be offensive. Toxicity classification in textual content remains a significant problem.

While differential privacy (DP) is a prominent method to gauge the degree of security provided to the models, its application is commonly limited to the model fine-tuning stage, due to the performance degradation when DP is applied during the pre-training stage. Consequently, DP is yet not capable of protecting a substantial portion of the data used during the initial pre-training process. In this work, we provide a theoretical understanding of the efficacy of DP training by analyzing the per-iteration loss improvement, through the lens of Hessian matrix for large neural networks. We make a key observation that DP optimizers' performance degradation can be significantly mitigated by the use of limited public data, which leads to a novel DP continual pre-training strategy. Empirically, using only 10% of public data and 90% of private data, our strategy can achieve DP accuracy of 41.5% on ImageNet-21k (with ϵ = 8), as well as non-DP accuracy of 55.7% and 60.0% on downstream tasks Places365 and iNaturalist-2021, respectively, on par with state-of-the-art standard pre-training and substantially outperforming existing DP pre-trained models. Our DP pre-trained models are released in fastDP library ( https://github.com/

Sparse training has received an upsurging interest in machine learning due to its tantalizing saving potential for the entire training process as well as inference.

Large language models (LLMs) demonstrate surprising capabilities, but we do not understand how they are implemented.

In this paper, we introduce a set of spatial-temporal logic rules as knowledge to explain human actions.

Unlike natural language processing and computer vision, the development of Foundation Models (FMs) for time series forecasting is blocked due to data scarcity.