The growing scale of machine learning applications has made data labeling costs a critical bottleneck in deploying ML systems [1, 2, 3]. Semi-supervised learning (SSL) addresses this challenge by leveraging unlabeled data alongside limited labeled examples [4]. Traditional SSL approaches like pseudo-labeling and consistency regularization have demonstrated strong performance across domains, particularly in computer vision and natural language processing [5, 6, 4]. Recent advances in foundation models have enabled zero-shot inference on novel tasks without taskspecific training [7, 8]. These models can generate predictions for unseen tasks by leveraging their pretrained knowledge, offering a promising direction for reducing labeling requirements. Several works have proposed integrating these zero-shot capabilities into SSL frameworks [9, 10]. Current approaches primarily use foundation models as teacher networks for generating pseudo-labels through inference, which requires complex model distillation and introduces additional training overhead.

We address the problem of Federated Learning (FL) where users are distributed and partitioned into clusters. This setup captures settings where different groups of users have their own objectives (learning tasks) but by aggregating their data with others in the same cluster (same learning task), they can leverage the strength in numbers in order to perform more efficient Federated Learning. We propose a new framework dubbed the Iterative Federated Clustering Algorithm (IFCA), which alternately estimates the cluster identities of the users and optimizes model parameters for the user clusters via gradient descent. We analyze the convergence rate of this algorithm first in a linear model with squared loss and then for generic strongly convex and smooth loss functions. We show that in both settings, with good initialization, IFCA converges at an exponential rate, and discuss the optimality of the statistical error rate. In the experiments, we show that our algorithm can succeed even if we relax the requirements on initialization with random initialization and multiple restarts. We also present experimental results showing that our algorithm is efficient in non-convex problems such as neural networks and outperforms the baselines on several clustered FL benchmarks created based on the MNIST and CIFAR-10 datasets by $5\sim 8\%$.