Probabilistic Bilevel Coreset Selection

These superior performances are mostly achieved The goal of coreset selection in supervised learning via learning from huge amounts of data. However, this datadriven is to produce a weighted subset of data, so that paradigm also poses several new challenges: 1) the training only on the subset achieves similar performance cumbersome dataset becomes harder to store and transfer; as training on the entire dataset. Existing 2) for some real applications, such as continual learning, methods achieved promising results in resourceconstrained one can only access a small number of training data at each scenarios such as continual learning stage of training; 3) in some more extreme scenarios, where and streaming. However, most of the existing algorithms the training data is incorrectly labelled, or they are collected are limited to traditional machine learning from different domains, more training data may even hurt models. A few algorithms that can handle the model's performance. To address these issues, a natural large models adopt greedy search approaches due idea is to select a small subset (i.e., coreset) comprised of to the difficulty in solving the discrete subset selection most informative training samples, such that training on this problem, which is computationally costly subset can achieve comparable or even better performance when coreset becomes larger and often produces with that on the full dataset, which is verified in Appendix suboptimal results. In this work, for the first time D. Therefore, how to construct a good coreset for DNNs we propose a continuous probabilistic bilevel formulation now becomes a crucial problem. of coreset selection by learning a probablistic weight for each training sample. The overall We notice that, coreset selection has been investigated for objective is posed as a bilevel optimization the traditional machine learning models, e.g., SVM (Tsang problem, where 1) the inner loop samples coresets et al., 2005), logistic regression (Huggins et al., 2016) and and train the model to convergence and 2) Gaussian mixture model (Lucic et al., 2017), for a long the outer loop updates the sample probability progressively time to accelerate the training process and lots of effective according to the model's performance.