Growing Winning Subnetworks, Not Pruning Them: A Paradigm for Density Discovery in Sparse Neural Networks

The lottery ticket hypothesis suggests that dense networks contain sparse subnet-works that can be trained in isolation to match full-model performance. Existing approaches--iterative pruning, dynamic sparse training, and pruning at initialization--either incur heavy retraining costs or assume the target density is fixed in advance. We introduce Path Weight Magnitude Product-biased Random growth (PWMPR), a constructive sparse-to-dense training paradigm that grows networks rather than pruning them, while automatically discovering their operating density. Starting from a sparse seed, PWMPR adds edges guided by path-kernel-inspired scores, mitigates bottlenecks via randomization, and stops when a logistic-fit rule detects plateauing accuracy. Experiments on CIFAR, TinyImageNet, and ImageNet show that PWMPR approaches the performance of IMP-derived lottery tickets--though at higher density--at substantially lower cost ( 1.5x dense vs. 3-4x for IMP). These results establish growth-based density discovery as a promising paradigm that complements pruning and dynamic sparsity. Their success has largely been driven by scaling: larger and denser models trained on larger datasets consistently improve performance (Kaplan et al., 2020). Y et this progress comes at immense computational cost, motivating the search for sparse alternatives that retain dense-level accuracy with reduced training and inference requirements. The lottery ticket hypothesis (L TH) (Frankle & Carbin, 2018) crystallized this challenge: dense networks contain sparse subnetworks ("winning tickets") that can be trained in isolation to match the full model's accuracy. Iterative Magnitude Pruning (IMP) demonstrates such subnetworks, but at prohibitive cost--often 3-4x more than dense training. This sparked extensive research into pruning-at-initialization (Lee et al., 2018; Tanaka et al., 2020), dynamic sparse training (Mocanu et al., 2018; Evci et al., 2020), and reparameterization methods (Mostafa & Wang, 2019; Kusupati et al., 2020).