To finish the derivation of Eq. 6, we need to compute E Tucker et al. 2017) and thus automatically choose the coupling which is favorable for the function Input images to the networks were centered with the global mean of the training dataset. The variance is measured based on 5000 Monte-Carlo samples at each iteration. In Appendix Figure 5, we compare gradient estimators for the toy problem Section 5.1, for which the REINFORCE LOO and ARM, especially as the problem becomes harder with increasing φ . We report the ELBO on training set (left column), the 100-sample bound on test set (middle column) and the variance of gradients (right column) for linear (top row) and nonlinear (bottom row) models. We report the ELBO on the training set (left), the 100-sample bound on the test set (middle), and the variance of the gradient estimator (right).

Averaging neural network parameters is an intuitive method for fusing the knowledge of two independent models. It is most prominently used in federated learning. If models are averaged at the end of training, this can only lead to a good performing model if the loss surface of interest is very particular, i.e., the loss in the midpoint between the two models needs to be sufficiently low. This is impossible to guarantee for the non-convex losses of state-of-the-art networks. For averaging models trained on vastly different datasets, it was proposed to average only the parameters of particular layers or combinations of layers, resulting in better performing models. To get a better understanding of the effect of layer-wise averaging, we analyse the performance of the models that result from averaging single layers, or groups of layers. Based on our empirical and theoretical investigation, we introduce a novel notion of the layer-wise linear connectivity, and show that deep networks do not have layer-wise barriers between them. In addition, we analyze layer-wise personalization averaging and conjecture that in particular problem setup all partial aggregations result in the approximately same performance. One of the multiple applications for such on Y-axis developed throughout training insights is, for example, knowledge fusion performed in a epochs on X-axis. First row shows the more efficient way than straightforward model ensembles.

It has been widely recognized that adversarial examples can be easily crafted to fool deep networks, which mainly root from the locally non-linear behavior nearby input examples. Applying mixup in training provides an effective mechanism to improve generalization performance and model robustness against adversarial perturbations, which introduces the globally linear behavior in-between training examples. However, in previous work, the mixup-trained models only passively defend adversarial attacks in inference by directly classifying the inputs, where the induced global linearity is not well exploited. Namely, since the locality of the adversarial perturbations, it would be more efficient to actively break the locality via the globality of the model predictions. Inspired by simple geometric intuition, we develop an inference principle, named mixup inference (MI), for mixup-trained models. MI mixups the input with other random clean samples, which can shrink and transfer the equivalent perturbation if the input is adversarial. Our experiments on CIFAR-10 and CIFAR-100 demonstrate that MI can further improve the adversarial robustness for the models trained by mixup and its variants.