Growing a Twig to Accelerate Large Vision-Language Models