From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning