Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models

Traditional knowledge distillation (KD) methods manually design student architectures to compress large models given pre-specified computational cost.