Avoiding spurious sharpness minimization broadens applicability of SAM