The Hardest Part

I'd like to thank Moritz, and Nisheeth, and Sanjeev for letting me guest post over at Off The Convex Path. I really enjoyed writing up my thoughts, so I've decided to dive in and try this for real. We're in the middle of a very exciting time in machine learning: the theory community is hungry to learn the fine details of practice, and the applied folks are looking for more insights into accelerating the training of large models. I'm sure many fascinating results are soon to come from these interactions, and this has motivated me to blog about the interface between theory and practice in optimization and machine learning. I'm going to start by following up on my last post, which ended with a vexing question… If saddle points are easy to avoid, then the question remains as to what exactly makes nonconvex optimization difficult? First, let me say that it's a bit ridiculous to define a class of problems using the "non-" prefix.