Machine learning models are being increasingly adopted to make decisions in a range of domains, such as finance, insurance, medical diagnosis, recruitment, and many more [2]. Therefore, we are often confronted with the need - sometimes imposed by regulatory bodies - to ensure that such machine learning models do not lead to decisions that discriminate individuals from a certain demographic group. The development of machine learning models that are fair across different (demographic) groups has been well studied in traditional learning setups where there is a single entity responsible for learning a model based on a local dataset holding data from individuals of the various groups. However, there are settings where the data representing different demographic groups is spread across multiple entities rather than concentrated on a single entity/server. For example, consider a scenario where various hospitals wish to learn a diagnostic machine learning model that is fair (or performs reasonably well) across different demographic groups but each hospital may only contain training data from certain groups because - in view of its geo-location - it serves predominantly individuals of a given demographic [5]. This new setup along with the conventional centralized one are depicted in Figure 1.