The optimization landscape of each task consists of a deep valley, a property that has been observed in neural network optimization landscapes [22], and the bottom ofeachvalleyischaracterized by high positive curvature and large differences in the task gradient magnitudes.

While the toy and scRNA-seq datasets do not have a split, we used the training and test set of CIFAR-10 jointly for the unsupervised UMAP dimensionreduction.

Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems. Neglecting the fundamental importance of data has given rise to inaccuracy, bias, and fragility in real-world applications, and research is hindered by saturation across existing dataset benchmarks.

Commonly used classification algorithms in machine learning, such as support vector machines, minimize a convex surrogate loss on training examples. In practice, these algorithms are surprisingly robusttoerrors inthe training data.

Self-supervised representation learning approaches haverecently surpassed their supervised learning counterparts on downstream tasks like object detection andimage classification.