The case against understanding why AI makes decisions

As deep-learning algorithms begin to set our life insurance rates and predict when we'll die, many AI experts are calling for more accountability around why those algorithms make the decisions they do. After all, if a self-driving car kills someone, we'd want to know what happened. But not everyone is sold on opening the "black box" of artificial intelligence. In a Medium post for Harvard's Berkman Klein Center, author and senior researcher David Weinberger writes that simplifying the processes deep-learning systems use to decide--a necessary step for humans to understand those processes--would actually undermine the reason we use algorithms in the first place: their complexity and nuance. "Human-constructed models aim at reducing the variables to a set small enough for our intellects to understand," Weinberger writes. "Machine learning models can construct models that work -- for example, they accurately predict the probability of medical conditions -- but that cannot be reduced enough for humans to understand or to explain them."