Federated Learning with Sample-level Client Drift Mitigation