Robust Federated Learning: The Caseof Affine Distribution Shifts