Combining Language Models For Specialized Domains: A Colorful Approach