On the Sensitivity of Reward Inference to Misspecified Human Models