Modern neural network architectures can leverage large amounts of data to generalize well within the training distribution.

In other words, they form a Simplex Equiangular Tight Frame (ETF).

Dataset distillation aims to learn a small synthetic dataset that preserves most of the information from the original dataset. Dataset distillation can be formulated as a bi-level meta-learning problem where the outer loop optimizes the meta-dataset and the inner loop trains a model on the distilled data.