Sparse training stands as a landmark approach in addressing the considerable training resource demands imposed by the continuously expanding size of Deep Neural Networks (DNNs). However, the training of a sparse DNN encounters great challenges in achieving optimal generalization ability despite the efforts from the state-of-the-art sparse training methodologies. To unravel the mysterious reason behind the difficulty of sparse training, we connect the network sparsity with neural loss functions structure, and identify the cause of such difficulty lies in chaotic loss surface. In light of such revelation, we propose $S^{2} - SAM$, characterized by a **S**ingle-step **S**harpness_**A**ware **M**inimization that is tailored for **S**parse training.

Neural networks augmented with external memory have the ability to learn algorithmic solutions to complex tasks.

Long-tailed visual recognition has received increasing attention recently.

We perform extensive experiments to confirm the effectiveness of AnyChange for zero-shot change detection.

By studying a two-layer CNN, we rigorously prove that SAM learns different features more uniformly, particularly in early epochs.

The framework relies on two methodological innovations.

Sharpness-aware minimization (SAM) has been shown to improve the generalization of neural networks.

This allows us to depict richer global behaviors of SAM.