Continually Learn to Map Visual Concepts to Large Language Models in Resource-constrained Environments