Dataset distillation aims to learn a small synthetic dataset that preserves most of the information from the original dataset. Dataset distillation can be formulated as a bi-level meta-learning problem where the outer loop optimizes the meta-dataset and the inner loop trains a model on the distilled data.

Single-index models are a class of functions given by an unknown univariate "link" function applied to an unknown one-dimensional projection of the input. These models are particularly relevant in high dimension, when the data might present low-dimensional structure that learning algorithms should adapt to. While several statistical aspects of this model, such as the sample complexity of recovering the relevant (one-dimensional) subspace, are well-understood, they rely on tailored algorithms that exploit the specific structure of the target function. In this work, we introduce a natural class of shallow neural networks and study its ability to learn single-index models via gradient flow . More precisely, we consider shallow networks in which biases of the neurons are frozen at random initialization. We show that the corresponding optimization landscape is benign, which in turn leads to generalization guarantees that match the near-optimal sample complexity of dedicated semi-parametric methods.

Method Y ear Family Train T est V alidation (HP/Model Selection) RandNet [11] 2017 AE ensemble Polluted =Train None, fixed - sensitivity analysis on some HPs RDA [48] 2017 AE Polluted =Train Best on Test, other HPs fixed DAGMM [49] 2018 AE & density Clean & Pol.d Disjoint None, fixed - sensitivity on reg. For MNIST, we choose Digit '4' and '5' For CIFAR10, we choose class'automobile' as the inlier-class. The inliers are labeled 0 and outliers are denoted as label 1. The data split and configuration are the same as described in the authors' provided code. With 4-to-8 different HPs each, the total number of configurations, and i.e. models trained, After the first-layer, the number of channels expand at rate of 2 .