Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network

One of the biggest issues in deep learning theory is the gener alization ability of networks with huge model size. The classical learning the ory suggests that overparameterized models cause overfitting. However, prac tically used large deep models avoid overfitting, which is not well explained by the c lassical approaches. To resolve this issue, several attempts have been made. Amon g them, the compression based bound is one of the promising approaches. However, the compression based bound can be applied only to a compressed network, and i t is not applicable to the non-compressed original network. In this paper, we gi ve a unified framework that can convert compression based bounds to those for n on-compressed original networks. The bound gives even better rate than the one for the compressed network by improving the bias term. By establishing the unified framework, we can obtain a data dependent generalization error bo und which gives a tighter evaluation than the data independent ones.