Training Task Experts through Retrieval Based Distillation