HARE: HumAn pRiors, a key to small language model Efficiency