Data-Aware Training Quality Monitoring and Certification for Reliable Deep Learning

Deep learning models have become crucial for tackling complex computational problems, owing to the rich representations they develop through their multi-layered structures and non-linear transformations [1, 2]. Despite their remarkable effectiveness, these models are often perceived as black boxes, raising concerns related to their robustness, reliability, and safety. As neural networks become increasingly integral to critical applications, ensuring that they are properly trained and perform as intended is paramount. To evaluate the training performance and a network's ability to store a model after training (i.e., achieve zero loss), one approach is to statistically analyze neural networks under certain assumptions. This has been done for networks with thresholding activation functions like ReLU, where researchers have determined the number of parameters needed to achieve full memory capacity [3]. It is well-known that for ReLU-based neural networks (NNs), once a sufficient number of weights is reached, the network can achieve full memory capacity or even zero loss in some cases. In [4], the authors theoretically demonstrate that in the over-parameterization regime, the stochastic gradient descent (SGD) algorithm can converge to the global minimum. However, these methods are statistical in nature and rely on specific assumptions about the input data and the model, which may limit their applicability. The first two authors contributed equally to this work.