Reviews: Modern Neural Networks Generalize on Small Data Sets

This paper presents an interesting idea, which is that deep neural networks are able to maintain reasonable generalization performance, even on relatively small datasets, because they can be viewed as an ensemble of uncorrelated sub-networks. Quality: The decomposition method seems reasonable, except for the requirement for the model and the sub-nets to achieve 100% training accuracy. While there are some datasets where this will be reasonable (often high-dimensional datasets), there are others where such an approach would work very badly. That seems to me a fundamental weakness of the approach, especially if there are datasets of that nature where deep neural nets still perform reasonably well. For a random forest, we have an unweighted combination of base classifiers, but it is a learned combination in the case of the decomposed sub-networks, and the weights are tuned on the training data.