To deepen our understanding of graph neural networks, we investigate the representation power of Graph Convolutional Networks (GCN) through the looking glass of graph moments, a key property of graph topology encoding path of various lengths. We find that GCNs are rather restrictive in learning graph moments. Without careful design, GCNs can fail miserably even with multiple layers and nonlinear activation functions. We analyze theoretically the expressiveness of GCNs, arriving at a modular GCN design, using different propagation rules. Our modular design is capable of distinguishing graphs from different graph generation models for surprisingly small graphs, a notoriously difficult problem in network science. Our investigation suggests that, depth is much more influential than width and deeper GCNs are more capable of learning higher order graph moments. Additionally, combining GCN modules with different propagation rules is critical to the representation power of GCNs.

This paper presents a theoretical analysis of the representational power of graph convolutional network (GCN) models. It is shown that these models are in many cases incapable of learning topological features of the graph such as graph moments, and these shortcomings are used to motivate new GCN architectures combining several propagation rules and incorporating multiple residual connections. The analysis seems to be sound (albeit distant to my area of expertise), and the obtained emprical results seem to support it. I believe that this paper could serve to better improve understanding of GCNs' representational power, and therefore I would vote for (marginal) acceptance. I have a few comments that could be used to improve the work: - In its current form, it is somewhat unclear what "GCN" exactly refers to.

Unfortunately reviewer_2 did not engage in the discussion even though it was needed, my recommendation is thus mainly based on the 3 other reviews. This paper investigates the expressive power of Graph neural networks by studying to which extent they can compute graph moments. This is an interesting and original approach and the theoretical findings are sound and relevant to the community. In preparing the camera-ready version of the paper, the authors should take the following points in consideration. The reviewers mentioned that the exposition of some of the contributions are somehow overstated and should be tuned down.

To deepen our understanding of graph neural networks, we investigate the representation power of Graph Convolutional Networks (GCN) through the looking glass of graph moments, a key property of graph topology encoding path of various lengths. We find that GCNs are rather restrictive in learning graph moments. Without careful design, GCNs can fail miserably even with multiple layers and nonlinear activation functions. We analyze theoretically the expressiveness of GCNs, arriving at a modular GCN design, using different propagation rules. Our modular design is capable of distinguishing graphs from different graph generation models for surprisingly small graphs, a notoriously difficult problem in network science.

To deepen our understanding of graph neural networks, we investigate the representation power of Graph Convolutional Networks (GCN) through the looking glass of graph moments, a key property of graph topology encoding path of various lengths. We find that GCNs are rather restrictive in learning graph moments. Without careful design, GCNs can fail miserably even with multiple layers and nonlinear activation functions. We analyze theoretically the expressiveness of GCNs, arriving at a modular GCN design, using different propagation rules. Our modular design is capable of distinguishing graphs from different graph generation models for surprisingly small graphs, a notoriously difficult problem in network science.