Similarly, self-supervised representation learning (SSL) is rapidly replacing supervised learning as the de-facto pretraining strategy for deep networks, due to improved scalability (unlabeled data is easier to collect) and generality (domain-specific SSL is often preferable to one-fits-all ImageNet pretraining [16,17]).

We show it depends on the precise way in which the limit is taken, and in particular on how the quantityofdata,thehiddenlayerwidth,&thelearningratescalesasd .

This beseenasminimizing26], which function [17]. Figure estimated (left) free but inhigh trib perform the Model-fr bors.

We initiate the study of stochastic optimization with oblivious noise, broadly generalizing the standard heavy-tailed noise setup.

MPNNs have been studied extensively in theory and practice [Böker et al., 2023, Gilmer et al., 2017,