Improved Generalization Bound of Permutation Invariant Deep Neural Networks

We theoretically prove that a permutation invariant property of deep neural networks largely improves its generalization performance. Learning problems with data that are invariant to permutations are frequently observed in various applications, for example, point cloud data and graph neural networks. Numerous methodologies have been developed and they achieve great performances, however, understanding a mechanism of the performance is still a developing problem. In this paper, we derive a theoretical generalization bound for invariant deep neural networks with a ReLU activation to clarify their mechanism. Consequently, our bound shows that the main term of their generalization gap is improved by n! where n is a number of permuting coordinates of data. Moreover, we prove that an approximation power of invariant deep neural networks can achieve an optimal rate, though the networks are restricted to be invariant. To achieve the results, we develop several new proof techniques such as correspondence with a fundamental domain and a scale-sensitive metric entropy. I NTRODUCTION A learning task with permutation invariant data frequently appears in various situations in data analysis. A typical example is learning on sets such as a point cloud, namely, the data are given as a set of points and permuting the points in the data does not change a result of its prediction. Another example is learning with graphs which contain a huge number of edges and nodes. Such the tasks are very common in various scientific fields [7, 8, 3], hence, numerous deep neural networks have been developed to handle such the data with invariance [15, 4, 11, 5, 13, 12].