Large Language Models as Visual Cross-Domain Learners