Synaptic Pruning: A Biological Inspiration for Deep Learning Regularization

Biological synaptic pruning removes weak neural connections to improve efficiency, while standard dropout in artificial networks randomly deactivates neurons without considering connection importance. We propose a magnitude-based synaptic pruning method that better emulates biological processes by gradually removing connections according to their contribution to model performance. Integrated directly into the training loop as a dropout replacement, our method computes weight importance from absolute magnitudes across layers and applies a cubic schedule to progressively increase global sparsity. At regular intervals, pruning masks are updated by thresholding weights, permanently removing low-importance connections while preserving gradient flow for active ones. This continuous, data-driven pruning removes the need for separate pruning and fine-tuning phases. We evaluated the method across multiple time series forecasting architectures, including Recurrent Neural Networks, Long Short-Term Memory, and Patch Time Series Transformer models, using four datasets. Our synaptic pruning approach achieved the best overall performance ranking across all architectures, with statistically significant improvements confirmed by Friedman tests ( p < 0. 01). In financial forecasting tasks, it reduced Mean Absolute Error by up to 20% compared to models using no dropout or standard dropout, with reductions reaching 52% in select transformer models. The proposed mechanism advances regularization by coupling dynamic weight elimination with progressive sparsification during training.