Data Science Fails: Fake News, Fake Data

Similarly, in the paper "Towards Debiasing Fact Verification Models," a related group of researchers identified a problem in the FEVER database that caused the unexpected result that claim-only classifiers performed competitively against evidence-aware classifiers. The authors identify strong cues for predicting labels solely based on the claims made. The FEVER dataset was constructed using crowdsourcing, allowing human bias to introduce human artifacts into the data. For example, the presence of negatively phrased statements was highly predictive of text flagged as fake news, yet this was not representative of fake news in the real world. Modern machine learning algorithms quickly identified these artifacts of human bias.