Unsupervised Federated Learning: A Federated Gradient EM Algorithm for Heterogeneous Mixture Models with Robustness against Adversarial Attacks

Unsupervised Federated Learning: A Federated Gradient EM Algorithm for Heterogeneous Mixture Models with Robustness against Adversarial AttacksYe Tian Haolei Weng Yang Feng Columbia University Michigan State University New York University Abstract While supervised federated learning approaches have enjoyed significant success, the domain of unsupervised federated learning remains relatively underexplored. In this paper, we introduce a novel federated gradient EM algorithm designed for the unsupervised learning of mixture models with heterogeneous mixture proportions across tasks. We begin with a comprehensive finite-sample theory that holds for general mixture models, then apply this general theory on Gaussian Mixture Models (GMMs) and Mixture of Regressions (MoRs) to characterize the explicit estimation error of model parameters and mixture proportions. Our proposed federated gradient EM algorithm demonstrates several key advantages: adaptability to unknown task similarity, resilience against adversarial attacks on a small fraction of data sources, protection of local data privacy, and computational and communication efficiency. 1 INTRODUCTION Federated learning (FDL) is a machine learning paradigm that allows the training of statistical models by leveraging data from various local tasks, while ensuring the data remains decentralized to protect privacy (Li et al., 2020a). Introduced a few years ago, notably by Google (Koneˇ cn` y et al., 2016; McMahan et al., 2017), FDL has witnessed remarkable success in a diverse range of applications, including smartphones (Hard et al., 2018), healthcare (Antunes et al., 2022), and the internet of things (Nguyen et al., 2021). In this paper, we delve into the realm of unsupervised FDL, a scenario in which each task involves a mixture of distributions. Before proceeding further, we summarize mathematical notations used in this paper here. For two positive sequences { a n} and { b n}, a n b n means a n/b n 0, a n b n or a n = O (b n) means a n/b n C <, and a n b n means a n/b n, b n/a n C < . For a random variable x n and a positive sequence a n, x n = O p(a n) means that for any ϵ > 0, there exists M > 0 such that sup n P( |x n/a n| > M) ϵ . For a vector x R d, x 2 represents its Euclidean norm. For any positive integer K, 1: K and [ K ] stand for the set { 1, 2, .. . And for any set S, |S | denotes its cardinality and S c denotes its complement.