Rethinking VLMs and LLMs for Image Classification