Efficient Multi-Prize Lottery Tickets: Enhanced Accuracy, Training, and Inference Speed

It is found that randomly-initialized dense networks contain subnetworks that can achieve test accuracy comparable to the trained dense network [5]. Based on this, the authors in [1] propose (and prove) a stronger Multi-Prize Lottery Ticket Hypothesis: A sufficiently over-parameterized neural network with random weights contains several subnetworks (winning tickets) that (a) have comparable accuracy to a dense target network with learned weights (prize 1), (b) do not require any further training to achieve prize 1 (prize 2), and (c) is robust to extreme forms of quantization (i.e., binary weights and/or activation) (prize 3). This provides a new paradigm for learning compact yet highly accurate binary neural networks simply by pruning and quantizing randomly weighted full precision neural networks. The proposed algorithm can find multi-prize tickets (MPTs) with SOTA accuracy for binary neural networks without ever updating the weight values.