This paper introduces the R package spca, which provides a computational framework for least squares sparse principal component analysis (LS-SPCA). Unlike other SPCA methods, LS-SPCA generates uncorrelated sparse principal components (sPCs) that effectively maximize the explained variance while maintaining strong correlations with standard principal components (PCs). The framework also includes more computationally efficient variants that produce mildly correlated sPCs, which often have lower cardinality while explaining equal or greater variance than the LS-SPCA optimal sPCs. The spca package is built on an efficient C++ backend for matrix computations, with distinct engines for tall and fat matrices, and a flexible R frontend. The user interface offers several options for computing sPCs, such as deciding whether sparsification should stop when a threshold for cumulative variance explained or R2 with the PCs is reached, and choosing between simple forward selection, stepwise forward selection, or backward elimination for variable selection. In addition to the print(), summary(), and plot() methods, the package includes tools for comparing different "spca" solutions, grouping sparse loadings, and representing foreign SPCA solutions as "spca" objects. This article demonstrates with real datasets the use of the package in a typical LS-SPCA workflow and briefly contrasts LS-SPCA with conventional SPCA solutions . Then it compares different LS-SPCA solutions obtained from the dataset. Finally, the performance of spca on large tall and fat matrices is discussed, showing that spca offers a computationally efficient alternative for computing interpretable sPCs.

We show for the first time that this leads both to improved convergence bounds under standard assumptions.

Corollary1statesthatf0 ismoreexpressivethan f when γ = 0, and thus the optimalf0 always outputs no higher loss, which, yet, is not true for arbitraryf0 (e.g.

Inspired by the robustness and efficiency of sparse representation in sparse coding based image restoration models, we investigate the sparsity of neurons in deep networks. Our method structurally enforces sparsity constraints upon hidden neurons. The sparsity constraints are favorable for gradient-based learning algorithms and attachable to convolution layers in various networks. Sparsity in neurons enables computation saving by only operating on non-zero components without hurting accuracy. Meanwhile, our method can magnify representation dimensionality and model capacity with negligible additional computation cost. Experiments show that sparse representation is crucial in deep neural networks for multiple image restoration tasks, including image super-resolution, image denoising, and image compression artifacts removal.

Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery tickets or sparse trainable subnetworks at initialization. This raises a foundational question: can we identify highly sparse trainable subnetworks at initialization, without ever training, or indeed without ever looking at the data? We provide an affirmative answer to this question through theory driven algorithm design. We first mathematically formulate and experimentally verify a conservation law that explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse, the premature pruning of an entire layer rendering a network untrainable. This theory also elucidates how layer-collapse can be entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow Pruning (SynFlow). This algorithm can be interpreted as preserving the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint. Notably, this algorithm makes no reference to the training data and consistently competes with or outperforms existing state-of-the-art pruning algorithms at initialization over a range of models (VGG and ResNet), datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to 99.99 percent). Thus our data-agnostic pruning algorithm challenges the existing paradigm that, at initialization, data must be used to quantify which synapses are important.

In dictionary selection, several atoms are selected from finite candidates that successfully approximate given data points in the sparse representation.

Current gradient-based NAS methods commonly ignore the constraint of sparsity in the search phase, but project the optimized solution onto a sparse one by post-processing.