Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers