Less is More for Improving Automatic Evaluation of Factual Consistency