An Impartial Take to the CNN vs Transformer Robustness Contest