Hybrid Conditional Gradient - Smoothing Algorithms with Applications to Sparse and Low Rank Regularization

Conditional gradient methods are old and well studied optimization algorithms. Their origin dates at least to the 50's and the Frank-Wolfe algorithm for quadratic programming [18] but they apply to much more general optimization problems. General formulations of conditional gradient algorithms have been studied in the past and various convergence properties of these algorithms have been proven. Moreover, such algorithms have found application in many fields, such as optimal control, statistics, signal processing, computational geometry and machine learning. Currently, interest in conditional gradient methods is undergoing a revival because of their computational advantages when applied to certain large scale optimization problems. Such examples are regularization problems involving sparsity or low rank constraints, which appear in many widely used methods in machine learning. Inspired by such algorithms, in this chapter we study a first-order method for solving certain convex optimization problems. We focus on problems of the form min {f(x) g(Ax) ω(x): x H}. 1 over a real Hilbert space H. We assume that f is a convex function with Hölder continuous gradient, g a Lipschitz continuous convex function, A a bounded linear operator and ω a convex function defined over a bounded domain.