For both learning rules, we prove overfitting is tempered.

Quite interestingly,modern deep networks havebeen known to be powerful enough to interpolate even randomized labels (Zhang et al., 2017; Liu et al., 2020), a phenomenon that is usually referred to asmemorization(Yunetal.,2019; Vershynin,2020;Bubeck etal.,2020), In this work, we lift the spherical requirement and offer an exponential improvementonthedependenceonδ.

We show that there are monotone data sets that cannot be interpolated by a monotone network of depth 2. On the other hand, we prove that for every monotone data set with n points in Rd, there exists an interpolating monotone network of depth 4 and size O(nd). Our interpolation result implies that every monotone function over [0,1]d can be approximated arbitrarily well by a depth4 monotone network, improving the previous best-known construction of depth d+1.

It is well known that modern deep neural networks are powerful enough to memorize datasets even when the labels have been randomized.

For both learning rules, we prove overfitting is tempered.

Monotone functions and data sets arise in a variety of applications. We study the interpolation problem for monotone data sets: The input is a monotone data set with n points, and the goal is to find a size and depth efficient monotone neural network with non negative parameters and threshold units that interpolates the data set. We show that there are monotone data sets that cannot be interpolated by a monotone network of depth 2. On the other hand, we prove that for every monotone data set with n points in R

Monotone functions and data sets arise in a variety of applications. We study the interpolation problem for monotone data sets: The input is a monotone data set with n points, and the goal is to find a size and depth efficient monotone neural network with non negative parameters and threshold units that interpolates the data set. We show that there are monotone data sets that cannot be interpolated by a monotone network of depth 2. On the other hand, we prove that for every monotone data set with n points in R

It is well known that modern deep neural networks are powerful enough to memorize datasets even when the labels have been randomized.