High-dimensional manifold of solutions in neural networks: insights from statistical physics