Mitigating the Problem of Strong Priors in LMs with Context Extrapolation