A Florida teenager taking a biology class at a community college got an upsetting note this year. A start-up called Honorlock had flagged her as acting suspiciously during an exam in February. She was, she said in an email to The New York Times, a Black woman who had been "wrongfully accused of academic dishonesty by an algorithm." What happened, however, was more complicated than a simple algorithmic mistake. It involved several humans, academic bureaucracy and an automated facial detection tool from Amazon called Rekognition.

When Amanda Kemper found out that artificial intelligence would help monitor students during her mechanical engineering class's final exam this summer, she was worried. Like many students, Kemper's classes at the University of Wisconsin-Madison shifted online suddenly in the spring due to the ongoing pandemic. With remote learning came remote exams: Starting in July, the university let instructors use software from Honorlock, which is one of numerous companies that can record video -- and much more -- of students as they take tests, and uses AI to point out any behavior that looks like cheating. Kemper learned about Honorlock a week before her final exam and she had a number of concerns. She didn't like the idea of being recorded and having that recording sent to her professor.