On Regularization Properties of Artificial Datasets for Deep Learning

In this paper, w e have presented analogies between the regularization methods for deep learning and data augmentation process interpreted as a noise injection. It was shown that, by generating the input data from high - level features, it is possible to regularize hidden layers of the netwo rk by exploiting the ability of deep networks to learn hierarchical representations . The analysis given here is theoretical, but there already are experimental results that partially confirm these observations . A case of convolutional neural networks for stenosis detection [14] have shown that pretraining the network on artificial dataset results in reduction of test error rate on real dataset, and, thus, smaller generalization gap. An improvement of test accuracy was also observed in the case of recurrent neural networks for ECG filtering, pretrained with synthetic signals [15] . A more definitive confirmation should be expected by the comparison of models trained for the same task with dataset s created by injecting noise either into input features or high - level features of the real data.