Not All Layers of LLMs Are Necessary During Inference