Saddles again

Thanks to Rong for the very nice blog post describing critical points of nonconvex functions and how to avoid them. I'd like to follow up on his post to highlight a fact that is not widely appreciated in nonlinear optimization. Though we often teach the contrary in our intro courses, it is in fact super hard to converge to a saddle point. If you move ever so slightly you fall off the saddle). Even simple algorithms like gradient descent with constant step sizes can't converge to saddle points unless you try really hard.