On the connections between algorithmic regularization and penalization for convex losses

However, penalization approach requires one to solve the problem (2) for a sequence of the tuning parameter λ to obtain an entire solution path, thus yielding a considerable computational burden. Efron et al. [2004] showed that the optimal solution path of Lasso is piecewise linear and proposed LARS algorithm to compute the full solution path of Lasso efficiently. This result w as extended to more generic cases by Rosset and Zhu [2007] who derived a general c haracterization of the properties of (loss f, penalty ψ) pairs giving piecewise linear coefficient paths that allow for efficient generation of the full regulari zed coefficient paths.