With the remarkable success achieved by Multimodal Large Language Models (MLLMs), numerous benchmarks have been designed to assess MLLMs' ability to guide their development in image perception tasks (e.g., image captioning and visual question answering). Moreover, these benchmarks contain many overly simple problems or challenging samples, which do not effectively differentiate the capabilities among various MLLMs. To address these challenges, we propose a pipeline to process the existing benchmarks, which consists of two modules (i.e., Semi-Automated Screening Process and Eliminating Answer Leakage). The Semi-Automated Screening Process filters out samples that cannot distinguish the model's capabilities by synthesizing various MLLMs and manually evaluating them. The Eliminate Answer Leakage filters the samples whose answers can be inferred without images. Finally, we curate the LIME-M: Less Is More for Evaluation of Multimodal LLMs, which is a lightweight Multimodal benchmark and can more effectively evaluate the performance of different models. Our experiments demonstrate that 1. LIME-M can better distinguish the performance of different MLLMs within less sample numbers (24% of original) and time spent (23% of original); 2. LIME-M eliminates answer leakage and mainly focuses on the information within images; 3. The current automatic metric (i.e., CIDEr) is insufficient for evaluating MLLMs' capabilities in captioning. Moreover, we find that removing the caption task score when calculating the overall score demonstrates a more precise reflection of model performance differences. In order to better understand the model's capabilities and guide to address the shortcomings of MLLMs, researchers have developed numerous benchmarks for various tasks (Antol et al., 2015; Wei et al., 2023; Fu et al., 2023; Yue et al., 2024; Wu et al., 2024a). These benchmarks thoroughly explore the capabilities of MLLMs in various tasks such as image captioning, image question answering, and Multimodal retrieving. However, existing MLLM benchmarks and unified evaluation frameworks cannot effectively and efficiently reflect the ability of MLLMs.