A Discrepancy-Based Perspective on Dataset Condensation

Deep learning has achieved remarkable success across a wide range of real-world applications, largely driven by the development of powerful neural network architectures (LeCun et al., 2015; Schmidhuber, 2015). In contrast to classical machine learning models, modern deep learning systems rely heavily on large and diverse datasets. However, they often suffer from limited interpretability and tend to underperform when trained on smaller datasets. Both theoretical and empirical studies suggest that increasing the size of both the dataset and the model generally leads to improved performance. This insight has motivated practitioners to scale up training datasets and neural network architectures to unprecedented levels in pursuit of higher accuracy (Kaplan et al., 2020; Hestness et al., 2017). However, such scaling comes at a cost: larger models require significantly more energy and contribute to the growing carbon footprint during both training and deployment (Strubell et al., 2020; Anthony et al., 2020; Patterson et al., 2021). Rather than solely pursuing ever-larger models, many studies advocate for the opposite direction: regularization and compression.