For the convenience of the reader, we summarize a list of notations blow. 1. null G In Appendix B.1, we present a general statement of Theorem 3.1 (a) along with its proof. Theorem 3.1 (a) states the order recovery guarantee for a specified parameter We summarize the bounds for (I) and (II) in Lemma B.1 and Lemma B.2, which can be found in Collecting the results in Lemma B.1 and Lemma B.2 and reorganizing the terms in the inequalities, we have the following conclusion. We now state the proof of this Lemma. Then we bound the first term using the concentration bound on Chi-squared random variables. For the non-identifiable models, we can use Lemma H.1 in a similar way to obtain that with probability We now state the proof of this Lemma.

We prove in this work that the well-known lasso problem can be solved exactly without homotopy using novel differential inclusions techniques. Specifically, we show that a selection principle from the theory of differential inclusions transforms the dual lasso problem into the problem of calculating the trajectory of a projected dynamical system that we prove is integrable. Our analysis yields an exact algorithm for the lasso problem, numerically up to machine precision, that is amenable to computing regularization paths and is very fast. Moreover, we show the continuation of solutions to the integrable projected dynamical system in terms of the hyperparameter naturally yields a rigorous homotopy algorithm. Numerical experiments confirm that our algorithm outperforms the state-of-the-art algorithms in both efficiency and accuracy. Beyond this work, we expect our results and analysis can be adapted to compute exact or approximate solutions to a broader class of polyhedral-constrained optimization problems.

Memory can be an issue when estimating a large-scale inverse covariance matrix. In the setting where an l^1 penalty is used, the authors propose a block approach that significantly reduces memory usage while satisfying convergence guarantees.

For the convenience of the reader, we summarize a list of notations blow. 1. null G In Appendix B.1, we present a general statement of Theorem 3.1 (a) along with its proof. Theorem 3.1 (a) states the order recovery guarantee for a specified parameter We summarize the bounds for (I) and (II) in Lemma B.1 and Lemma B.2, which can be found in Collecting the results in Lemma B.1 and Lemma B.2 and reorganizing the terms in the inequalities, we have the following conclusion. We now state the proof of this Lemma. Then we bound the first term using the concentration bound on Chi-squared random variables. For the non-identifiable models, we can use Lemma H.1 in a similar way to obtain that with probability We now state the proof of this Lemma.

Abstract--The adaptive Iterative Soft-Thresholding Algorithm (IST A) has been a popular algorithm for finding a desirable solution to the LASSO problem without explicitly tuning the regularization parameter λ. Despite that the adaptive IST A is a successful practical algorithm, few theoretical results exist. In this paper, we present the theoretical analysis on the adaptive IST A with the thresh-olding strategy of estimating noise level by median absolut e deviation. We show properties of the fixed points of the algorithm, including scale equivariance, non-uniqueness, and local stability, prove the local linear convergence guarantee, and show its global convergence behavior . Many sparse approximation problems in machine learning and signal processing can be obtained as the solution to the LASSO problem, which can be solved by IST A. Despite its popularity, tuning The obtained LASSO solution is optimal in the mean-squared-error (MSE) sense with minimum assumptions, but LARS is not competitive in terms of computation time for large-scale problems [7].

Summary: This paper presents an unprecedentedly fast method for eliminating variables during iterations of multi-task group lasso that is provably safe, meaning that all variables that are eliminated would ultimately obtain zero weights when running vanilla group lasso. This method is iterative; as the primal solver converges, it eliminates an increasing number of variables. The authors compare their method to previous methods and demonstrate that all previous methods are either unsafe or are substantially slower for small duality gaps. The authors describe how their method should be applied to specific cases of group lasso, including l1 and l1/l2 regularized logistic regression, and present real applications where their method achieves a substantial speed-up over vanilla group lasso and an existing method for small duality gap thresholds. The primary situation where this method obtains substantial speed improvements over alternatives is where the duality gap threshold is extremely small.

We show that training deep neural networks (DNNs) with absolute value activation and arbitrary input dimension can be formulated as equivalent convex Lasso problems with novel features expressed using geometric algebra. This formulation reveals geometric structures encoding symmetry in neural networks. Using the equivalent Lasso form of DNNs, we formally prove a fundamental distinction between deep and shallow networks: deep networks inherently favor symmetric structures in their fitted functions, with greater depth enabling multilevel symmetries, i.e., symmetries within symmetries. Moreover, Lasso features represent distances to hyperplanes that are reflected across training points. These reflection hyperplanes are spanned by training data and are orthogonal to optimal weight vectors. Numerical experiments support theory and demonstrate theoretically predicted features when training networks using embeddings generated by Large Language Models. Recent advancements have demonstrated that deep neural networks are powerful models that can perform tasks including natural language processing, synthetic data and image generation, classification, and regression. However, research literature still lacks in intuitively understanding why deep networks are so powerful: what they "look for" in data, or in other words, how each layer extracts features. We are interested in the following question: Is there a fundamental difference in the nature of functions learned by deep networks, as opposed to shallow networks? We answer this question by transforming non-convex training problems into convex formulations and analyzing their structure.

Lasso is a widely used regression technique to find sparse representations. When the dimension of the feature space and the number of samples are extremely large, solving the Lasso problem remains challenging. To improve the efficiency of solving large-scale Lasso problems, El Ghaoui and his colleagues have proposed the SAFE rules which are able to quickly identify the inactive predictors, i.e., predictors that have 0 components in the solution vector. Then, the inactive predictors or features can be removed from the optimization problem to reduce its scale. By transforming the standard Lasso to its dual form, it can be shown that the inactive predictors include the set of inactive constraints on the optimal dual solution.

The Lasso is a cornerstone of modern multivariate data analysis, yet its performance suffers in the common situation in which covariates are correlated.