Divide-or-Conquer? Which Part Should You Distill Your LLM?