Typical and atypical solutions in non-convex neural networks with discrete and continuous weights