In theory, the choice of ReLU0(0) in [0,1] for a neural network has a negligible influence both on backpropagation and training. Yet, in the real world, 32 bits default precision combined with the size of deep learning problems makes it a hyperparameter of training methods. We investigate the importance of the value of ReLU0(0) for several precision levels (16, 32, 64 bits), on various networks (fully connected, VGG, ResNet) and datasets (MNIST, CIFAR10, SVHN, ImageNet). We observe considerable variations of backpropagation outputs which occur around half of the time in 32 bits precision. The effect disappears with double precision, while it is systematic at 16 bits. For vanilla SGD training, the choice ReLU0(0) = 0 seems to be the most efficient. For our experiments on ImageNet the gain in test accuracy over ReLU0(0) = 1 was more than 10 points (two runs). We also evidence that reconditioning approaches as batch-norm or ADAM tend to buffer the influence of ReLU0(0)'s value. Overall, the message we convey is that algorithmic differentiation of nonsmooth problems potentially hides parameters that could be tuned advantageously.

We investigate a stochastic counterpart of majority votes over finite ensembles of classifiers, and study its generalization properties. While our approach holds for arbitrary distributions, we instantiate it with Dirichlet distributions: this allows for a closed-form and differentiable expression for the expected risk, which then turns the generalization bound into a tractable training objective. The resulting stochastic majority vote learning algorithm achieves state-of-the-art accuracy and benefits from (non-vacuous) tight generalization bounds, in a series of numerical experiments when compared to competing algorithms which also minimize PACBayes objectives - both with uninformed (data-independent) and informed (datadependent) priors.

A.1 Comparison with SOTAModels on 60%/20%/20% Random Splits The main results of the full sets of experiments 17 with statistics of datasets are summarized in Table 2, where we report the mean accuracy (%) and standard deviation. We can see that after applied in ACM or ACMII framework, the performance of baseline models are boosted on almost all tasks and achieve SOTA performance on 9out of 10datasets. Especially, ACMII-GCN+ performs the best in terms of average rank (4.40) across all datasets. Overall, It suggests that ACM or ACMII framework can significantly increase the performance of GNNs on node classification tasks on heterophilic graphs and maintain highly competitive performance on homophilic datasets. The best results are highlighted in grey and the best baseline results (SOTA in Figure 6) are underlined. Results "*" are reported from [8, 26] and results " " are from [36].

Graph Neural Networks (GNNs) extend basic Neural Networks (NNs) by using graph structures based on the relational inductive bias (homophily assumption). While GNNs have been commonly believed to outperform NNs in real-world tasks, recent work has identified a non-trivial set of datasets where their performance compared to NNs is not satisfactory. Heterophily has been considered as the main cause of this empirical observation and numerous works have been put forward to address it. In this paper, we first revisit the widely used homophily metrics and point out that their consideration of only graph-label consistency is a shortcoming. Then, we study heterophily from the perspective of post-aggregation node similarity and define new homophily metrics, which are verified to be advantageous compared to existing ones. Based on this investigation, we prove that some harmful cases of heterophily can be effectively addressed by local diversification operation. Then, we propose the Adaptive Channel Mixing (ACM), a framework to adaptively exploit aggregation, diversification and identity channels node-wisely to extract richer localized information for diverse node heterophily situations. ACM is more powerful than the commonly used uni-channel framework for node classification tasks on heterophilic graphs and is easy to be implemented in baseline GNN layers. When evaluated on 10 benchmark node classification tasks, ACM-augmented baselines consistently achieve significant performance gain, exceeding state-of-theart GNNs on most tasks without incurring significant computational burden.

We investigate the parameterized complexity of Bayesian Network Structure Learn-1 ing (BNSL), a classical problem that has received significant attention in empirical2 but also purely theoretical studies. We follow up on previous works that have3 analyzed the complexity of BNSL w.r.t. the so-called superstructure of the input.4 While known results imply that BNSL is unlikely to be fixed-parameter tractable5 even when parameterized by the size of a vertex cover in the superstructure, here we6 show that a different kind of parameterization--notably by the size of a feedback7 edge set--yields fixed-parameter tractability. We proceed by showing that this8 result can be strengthened to a localized version of the feedback edge set, and9 provide corresponding lower bounds that complement previous results to provide a10 complexity classification of BNSL w.r.t.