Analytic Insights into Structure and Rank of Neural Network Hessian Maps Sidak Pal Singh

This yields exact formulas and tight upper bounds for the Hessian rank of deep linear networks -- allowing for an elegant interpretation in terms of rank deficiency. Moreover, we demonstrate that our bounds remain faithful as an estimate of the numerical Hessian rank, for a larger class of models such as rectified and hyperbolic tangent networks.