Merging of neural networks

The final result is quite sensitive to the starting random seed as reported in [1, 2], who observed 0.5% difference in accuracy between worst and best seed on Imagenet dataset and 1.8% difference on CIFAR-10 dataset. Thus, one might need to run an experiment several times to avoid hitting the unlucky seed. The final selected network is just the one with the best validation accuracy. We believe that discrepancy between starting seeds performance can be explained by selecting slightly different features in hidden layers in each initialization. One might ask a question, can we somehow select better features for network training? One approach is to train a bigger network and then select the most important channels via channel pruning [3, 4, 5, 6]. Training a big network, which is subsequently pruned, might be in many cases prohibitive, since increasing network width by a factor of two results in a four times increase in FLOPs and also might require a change in some hyperparameters (e.g.