Enhancing Robust Fairness via Confusional Spectral Regularization