Surprising Efficacy of Fine-Tuned Transformers for Fact-Checking over Larger Language Models