Specialized Language Models with Cheap Inference from Limited Domain Data