Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supply

Traditional end-to-end (E2E) training of deep networks necessitates storing intermediate activations for back-propagation, resulting in a large memory footprint on GPUs and restricted model parallelization. As an alternative, greedy local learning partitions the network into gradient-isolated modules and trains supervisely based on local preliminary losses, thereby providing asynchronous and parallel training methods that substantially reduce memory cost. However, empirical experiments reveal that as the number of segmentations of the gradient-isolated module increases, the performance of the local learning scheme degrades substantially, severely limiting its expansibility. To avoid this issue, we theoretically analyze the greedy local learning from the standpoint of information theory and propose a ContSup scheme, which incorporates context supply between isolated modules to compensate for information loss. CIFAR, SVHN, STL-10) achieve SOTA results and indicate that our proposed method can significantly improve the performance of greedy local learning with minimal memory and computational overhead, allowing for the boost of the number of isolated modules. Our codes are available at https://github.com/Tabct/ContSup. End-to-end (E2E) back-propagation, a standard training paradigm for deep neural networks, enables deep neural networks to solve complex tasks and cognitive applications with great success (Szegedy et al., 2015; He et al., 2016; Huang et al., 2016). As shown in Figure 1a, an E2E training loss is calculated at the final layer, and the error is propagated backward layer-by-layer for weights update. In this case, the E2E is caught in the well-known backward-locking problem (Jaderberg et al., 2017; Frenkel et al., 2021; Duan & Principe, 2022), which prohibits module updates until all dependent modules have completed forward and backward passes, and restricts the network from performing training in a sequential manner (Jaderberg et al., 2017).