Robust Federated Learning: The Case of Affine Distribution Shifts