Semi-Variance Reduction for Fair Federated Learning