Combating Exacerbated Heterogeneity for Robust Models in Federated Learning