Often in statistics and machine learning we are required to partition a dataset, e.g., when (i) splitting a dataset for training and testing, (ii) subsampling from Big Data for conducting tractable statistical analysis or to save storage space, (iii) generating multiple splits of a dataset for divide-and-conquer procedures to act upon, and (iv) creating k-fold cross validation sets for model tuning and validation. For this purpose, we propose a novel method named Twinning that can be used for partitioning a dataset into statistically similar sets. Twinning is motivated from the recent work on optimal data splitting for model validation, by Joseph and Vakayil (2021). For model validation, the common practice is to randomly split the dataset into training and testing sets, e.g., for an 80-20 split, 20% of the dataset is selected randomly for testing, while the remaining 80% is used for training the model. It is easy to see that such random splitting can plausibly give rise to pathological splits, wherein the training and testing sets cover roughly disjoint regions of the feature space, thereby resulting in poor testing performance of the model.

In this paper, we analyze a number of architectural features of Deep Neural Networks (DNNs), using the so-called Neural Tangent Kernel (NTK). The NTK describes the training trajectory and generalization of DNNs in the infinite-width limit. In this limit, we show that for (fully-connected) DNNs, as the depth grows, two regimes appear: "freeze" (also known as "order"), where the (scaled) NTK converges to a constant (slowing convergence), and "chaos", where it converges to a Kronecker delta (limiting generalization). We show that when using the scaled ReLU as a nonlinearity, we naturally end up in the "freeze". We show that Batch Normalization (BN) avoids the freeze regime by reducing the importance of the constant mode in the NTK. A similar effect is obtained by normalizing the nonlinearity which moves the network to the chaotic regime. We uncover the same "freeze" and "chaos" modes in Deep Deconvolutional Networks (DC-NNs). The "freeze" regime is characterized by checkerboard patterns in the image space in addition to the constant modes in input space. Finally, we introduce a new NTK-based parametrization to eliminate border artifacts and we propose a layer-dependent learning rate to improve the convergence of DC-NNs. We illustrate our findings by training DCGANs using our setup. When trained in the "freeze" regime, we see that the generator collapses to a checkerboard mode. We also demonstrate numerically that the generator collapse can be avoided and that good quality samples can be obtained, by tuning the nonlinearity to reach the "chaos" regime (without using batch normalization).