FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity

We introduce two distinct types of network pruning within our study: 1) global pruning, which extends from server to client, and 2) local pruning, where each client's network is pruned based on its own specific data. In our setting, we assume federated pruning is the scenario with both possible global and local pruning. Federated network pruning, a closely related field, pursues the objective of identifying the optimal or near-optimal pruned neural network at each communication from the server to the clients, as documented in works of Jiang et al. (2022a) and Huang et al. (2022), for example. During the initial phase of global pruning, (Jiang et al., 2022a) isolates a single potent and reliable client to initiate model pruning. The subsequent stage of local pruning incorporates all clients, advancing the adaptive pruning process. This process involves not only parameter removal but also the reintroduction of parameters, complemented by the standard FedAvg (McMahan et al., 2017). However, the need for substantial local memory to record the updated relevance measures of all parameters in the full-scale model poses a challenge. As a solution to this problem, Huang et al. (2022) proposes an adaptive batch normalization and progressive pruning modules that utilize sparse local computation. Yet, these methods overlook explicit considerations for constraints related to client-side computational resources and communication bandwidth.