Deep Learning
Representing Long-Range Context for Graph Neural Networks with Global Attention
Graph neural networks are powerful architectures for structured datasets. However, current methods struggle to represent long-range dependencies. Scaling the depth or width of GNNs is insufficient to broaden receptive fields as larger GNNs encounter optimization instabilities such as vanishing gradients and representation oversmoothing, while pooling-based approaches have yet to become as universally useful as in computer vision.
A Theory
In this section, we provide proofs and additional details for Section 3. A.1 Norm constraint: total vs. individual We begin with a formal derivation of the formulas in Section 3.1. Then the following results hold: 1. ฮท < The above formulation allegedly lacks the third (divergent) regime. For the second statement, based on eq. A.4 More formally on the results of Section 3.2 In this section, we provide a more formal argument on the results of Section 3.2. According to the results of Section 3.1, solving it with the projected gradient method Here we provide additional plots depicting the behavior of individual ELRs in the toy example at the end of Section 3.2.