Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs).

Neural Information Processing Systems http://nips.cc/

From Figure 1(a), learning multiple linear sub-models and averaging the predictions (ensemble) is still a linear model, so it cannot tackleXOR problem. We compare the training cost of all methods from the two aspects;1). Thesub-model training enables themost adversarial attacks ofsub-models could be successfully defended. In particular, we train two kinds of models to defend against the attacks: 1). FromFigure2(a)and2(b),when0.01 ϵ 0.04, SoE without the collaboration training achieves a similar robustness compared with SoE.

Frankle et al.[11] proposed an iterative pruning with rewinding to avoid lower-stability phase to SGDnoiseintheearlytraining ineachIMPstep.

We adopt the natural assumption that the empirical NTK develops a block structure aligned with the class labels, i.e., samples within the same class have stronger

S4.2 Training Table S2 presents architectures.