TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models