Discrete and Continuous Deep Residual Learning Over Graphs

Pedro H.C. Avelar Anderson R. Tavares Marco Gori † Luis C. Lamb Abstract In this paper we propose the use of continuous residual modules for graph kernels in Graph Neural Networks. We show how both discrete and continuous residual layers allow for more robust training, being that continuous residual layers are those which are applied by integrating through an Ordinary Differential Equation (ODE) solver to produce their output. We experimentally show that these residuals achieve better results than the ones with non-residual modules when multiple layers are used, mitigating the low-pass filtering effect of GCN-based models. Finally, we apply and analyse the behaviour of these techniques and give pointers to how this technique can be useful in other domains by allowing more predictable behaviour under dynamic times of computation. 1 Introduction Graph Neural Networks (GNNs) are a promising framework to combine deep learning models and symbolic reasoning. Whereas conventional deep learning models, such as Convolutional Neural Networks (CNNs), effectively handle data represented in euclidean space, such as images, GNNs generalise their capabilities to handle non-Euclidean data, such as relational data with complex relationships and interdependencies between entities. Recently, deep learning techniques such as pooling, dynamic times of computation, attention, and adversarial training, which advanced the state-of-the-art in conventional deep learning (e.g. in CNNs), have been investigated in GNNs as well [1, 15, 26, 30]. Discrete residual modules, whose learned kernels are discrete derivatives over their inputs, have been proven effective to improve convergence and reduce the parameter space on CNNs, surpassing the state-of-the-art in image classification and other applications [11]. Given their effectiveness, the technique has been applied in many different areas and meta-models of deep learning to improve convergence and reduce the parameter space.