Libraries of hidden layer activity patterns can lead to better understanding of operating principles of deep neural networks

Deep neural networks (DNNs) can outperform human brains in specific tasks, but due to their lack of transparency in decision-making processes, it remains uncertain whether we can completely rely on DNNs when it comes to high-risk problems requiring rigorous decisions. As DNNs' operations rely on a massive number of linear/nonlinear computations in both parallel and sequential, it is impossible to distinguish every factor influencing their decision. Further, DNNs cannot be empirically tested in every potential condition. Therefore, we should consider the possibility that DNNs' decisions can be corrupted by unexpected factors; for instance, adversarial attacks can deceive DNNs quite effectively. Before DNNs are deployed to solve high-stakes problems, such vulnerability must be overcome. In our study, we propose an algorithm to provide better insights into DNNs' decision-making processes. Our experiments suggest that this algorithm can effectively trace DNNs' decision processes on a layer-by-layer basis and be used to detect adversarial attacks.