Inference Optimal VLMs Need Only One Visual Token but Larger Models