On the Limits of Applying Graph Transformers for Brain Connectome Classification

However, it did not produce improvements in performance or other notable benefits, see Appendix 1. For the Exphormer, we experimented with different numbers of layers, dropout rates for the network and attention mechanism, and numbers of attention heads. The final configuration used dropout probability of 0.1, attention dropout of 0.3, 2 layers, and 4 attention heads. All experiments used learning rate decay starting at 0.001, decaying by 1e 5, over a total of 100 epochs with 5 warmup epochs. We used three different seeds for both the Exphormer and ResidualGCN and assessed the alignment with the results in Said et al. (2023), which only included one run for each experiment. Apart from evaluating performance, we investigated potential advantages of using attention-based models. Our hypothesis was that the attention mechanism could enhance robustness to data noise, particularly in scenarios where certain graph structure components, such as edges, are missing. To verify that the graph structure, nodes and edges taken together, convey meaningful information for prediction, it is important to compare models under noisy or incomplete data settings. We simulate noisy incomplete data by removing edges based on a pre-specified probability of edge removal.