Amazon's sexist AI recruiting tool: how did it go so wrong?

Last year, Reuters broke the news that Amazon had been working on a secret AI recruiting tool that showed bias against women. I found it interesting as a case study of an AI project with broad implications for business people and machine learning professionals. After all, everyone has either hired or been hired at least once. We all have a stake in the recruiting game. Sadly, most reporting was sensationalist trash, and the news cycle quickly moved on. It seems nobody tried to answer the question of how a company of the caliber of Amazon -- with seemingly infinite resources -- could stumble so badly. Is AI technology inherently evil? Are all software engineers and data scientists sexist brutes? Is the technology too immature for complex business problems? Or is there something specific about AI projects that makes them difficult -- even for the best companies? The Reuter article was widely circulated and became a prime example of the pitfalls of AI projects.