Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition