We propose an innovative algorithm for non-convex composite federated learning that decouples the proximal operator evaluation and the communication between server and clients. Moreover, each client uses local updates to communicate less frequently with the server, sends only a single d-dimensional vector per communication round, and overcomes issues with client drift. In the analysis, challenges arise from the use of decoupling strategies and local updates in the algorithm, as well as from the non-convex and non-smooth nature of the problem. We establish sublinear and linear convergence to a bounded residual error under general non-convexity and the proximal Polyak-Lojasiewicz inequality, respectively. In the numerical experiments, we demonstrate the superiority of our algorithm over state-of-the-art methods on both synthetic and real datasets.

We introduce a locally differentially private (LDP) algorithm for online federated learning that employs temporally correlated noise to improve utility while preserving privacy. To address challenges posed by the correlated noise and local updates with streaming non-IID data, we develop a perturbed iterate analysis that controls the impact of the noise on the utility. Moreover, we demonstrate how the drift errors from local updates can be effectively managed for several classes of nonconvex loss functions. Subject to an $(\epsilon,\delta)$-LDP budget, we establish a dynamic regret bound that quantifies the impact of key parameters and the intensity of changes in the dynamic environment on the learning performance. Numerical experiments confirm the efficacy of the proposed algorithm.

With the rapid advancement of deep neural networks (DNNs), distributed training has become the mainstream approach for efficiently scaling up models. One of the most popular algorithms used in data-parallel training is All-Reduce (Li et al., 2020), known for its simplicity and its ability to maintain consistency However, All-Reduce training relies on high-speed network connections and homogeneous computational devices to ensure its efficiency (Zhang et al., 2020; Tandon et al., 2017). Decentralized algorithms, which originally gained attention in the fields of consensus algorithms (Johansson et al., 2007; Shi et al., 2015) and privacy-preserving techniques (Y an et al., 2012), have recently been explored as alternatives to All-Reduce in distributed training, especially in We believe there are several reasons for this: (1) Simply combining the best of each line of work does not necessarily lead to an effective overall system. We propose a simple yet accurate runtime model that quantifies key environmental parameters and estimates potential speedups. We design and analyze a decentralized variant of the Adam optimizer.

Many machine learning tasks, such as principal component analysis and low-rank matrix completion, give rise to manifold optimization problems. Although there is a large body of work studying the design and analysis of algorithms for manifold optimization in the centralized setting, there are currently very few works addressing the federated setting. In this paper, we consider nonconvex federated learning over a compact smooth submanifold in the setting of heterogeneous client data. We propose an algorithm that leverages stochastic Riemannian gradients and a manifold projection operator to improve computational efficiency, uses local updates to improve communication efficiency, and avoids client drift. Theoretically, we show that our proposed algorithm converges sub-linearly to a neighborhood of a first-order optimal solution by using a novel analysis that jointly exploits the manifold structure and properties of the loss functions. Numerical experiments demonstrate that our algorithm has significantly smaller computational and communication overhead than existing methods.

We propose a novel differentially private algorithm for online federated learning that employs temporally correlated noise to improve the utility while ensuring the privacy of the continuously released models. To address challenges stemming from DP noise and local updates with streaming noniid data, we develop a perturbed iterate analysis to control the impact of the DP noise on the utility. Moreover, we demonstrate how the drift errors from local updates can be effectively managed under a quasi-strong convexity condition. Subject to an $(\epsilon, \delta)$-DP budget, we establish a dynamic regret bound over the entire time horizon that quantifies the impact of key parameters and the intensity of changes in dynamic environments. Numerical experiments validate the efficacy of the proposed algorithm.

We propose a novel algorithm for solving the composite Federated Learning (FL) problem. This algorithm manages non-smooth regularization by strategically decoupling the proximal operator and communication, and addresses client drift without any assumptions about data similarity. Moreover, each worker uses local updates to reduce the communication frequency with the server and transmits only a $d$-dimensional vector per communication round. We prove that our algorithm converges linearly to a neighborhood of the optimal solution and demonstrate the superiority of our algorithm over state-of-the-art methods in numerical experiments.

The design This paper proposes a locally differentially private federated of DP algorithms in federated learning depends on the attack learning algorithm for strongly convex but possibly nonsmooth scenario and can be roughly divided into global DP and problems that protects the gradients of each worker local DP (LDP) [6, 7]. Global DP resists passive attackers against an honest but curious server. The proposed algorithm from outside the system and typically relies on the server to adds artificial noise to the shared information to ensure add noise to the aggregated information while the workers privacy and dynamically allocates the time-varying noise upload their true models or gradients, assuming that the upload variance to minimize an upper bound of the optimization communication channel is secure and the server is trustworthy error subject to a predefined privacy budget constraint.