We first invoke the following lemma which relates the empirical and true cumulative distribution functionsofi.i.d. Steps 1 2 4 8 16 Noregularization 4.17 4.04 4.05 4.04 4.05 L1(λ=10.0) In Figure B.1, we show additional visualizations of regression tasks. C.2 Classification Task Learner The task learner was ResNet-18 [3] for the SVHN and CIFAR-10 datasets, and anMLP with one hidden layer of500nodes and ReLU nonlinearities. TheCNNarchitecture was the4-layer convolutional net in [6] when the task learner was an MLP, and was ResNet-18 otherwise.

While various complexity measures for deep neural networks exist, specifying an appropriate measure capable of predicting and explaining generalization in deep networks has provenchallenging.

In this work, we initiate a theoretical study of the problem above.

Ourmethodologyincorporates a certain amount of backtracking when needed, allowing changes in causally preceding variables tominimize deviations from realistic scenarios. Specifically, we introduce a novel optimization framework that permits but also controls the extent of backtracking with a "naturalness" criterion. Empirical experiments demonstrate the effectiveness of our method.