Numerical influence of ReLU'(0) on backpropagation Jérôme Bolte IRT Saint Exupéry Toulouse School of Economics ISAE-SUPAERO Université Toulouse 1 Capitole ANITI ANITI Toulouse, France

Jan-21-2025, 02:24:34 GMT–Neural Information Processing Systems 

Yet, in the real world, 32 bits default precision combined with the size of deep learning problems makes it a hyperparameter of training methods.