Appendix A CommonsenseQA Error Patterns Throughout our experiments, we came across a variety of interesting failure cases for commonse

One key failure case was answers in the form of "the answer must be something that is ." Q: What might someone get from learning about science? Answer Choices: (a) headache (b) see things differently (c) increased knowledge (d) accidents (e) appreciation of nature A: The answer must be something that someone would get from learning about science. Learning about science would increase knowledge. Therefore, the answer is increased knowledge (c). These rationales, while perhaps useful to the model, read to us as opaque and unexplanatory. Sometimes the model will imply the answer that it has chosen in its question. Q: The christian rock band was on a large stage, the singer's foot got caught on a wire and he had a small what? Answer Choices: (a) commit sin (b) fail (c) disobey (d) stumble (e) boo boo A: The answer must be something that would cause the singer to stumble. The singer's foot got caught on a wire, so he would stumble.