Understanding the Role of LLMs in Multimodal Evaluation Benchmarks