Despite its theoretical appeal, this viewpoint lacks a crucial ingredient of deep learning in finite DNNs, laying at the heart of their success --feature learning. Here we consider DNNs trained with noisy gradient descent on a large training set and derive a self-consistent Gaussian Process theory accounting forstrongfinite-DNN and feature learning effects.

We can verify that all of above complexity (nearly) matches the corresponding lower bounds.

There are mainly two types of domain adaptation formulas:covariate shift[44, 37, 29, 13] and label shift [27, 2, 1], while we focus on the former in this paper since it appears more natural in recognition tasks and attracts more attention in the literature.

Tobespecific, our 10 meta-testdatasetsinclude Histology, Drawing, Dessert, Chinese Characters, Speed Limit Signs, Alienvs Predator, Gemstones, and Dog Breeds. Thusweuse Mean Squared Error (MSE) scores.