The Responsibility Quantification (ResQu) Model of Human Interaction with Automation

Abstract--Advanced automation is involved in information collection and evaluation, in decision-making and in the implementation of chosen actions. In such systems, human responsibility becomes equivocal, and there may exist a responsibility gap. Understanding human responsibility is particularly important when systems can harm people, as with autonomous vehicles or, most notably, with Autonomous Weapon Systems (AWS). Using Information Theory, we develop a responsibility quantification (ResQu) model of human interaction in automated systems and demonstrate its applications on decisions involving AWS. The analysis reveals that human comparative responsibility is often low, even when major functions are allocated to the human. Thus, broadly stated policies of keeping humans in the loop and having meaningful human control are misleading and cannot truly direct decisions on how to involve humans in advanced automation. Our responsibility model can guide system design decisions and can aid policy and legal decisions regarding human responsibility in highly automated systems. Financial markets largely function through algorithmic trading mechanisms [1, 2], semiconductor manufacturing is almost entirely automated [3], and decision support systems and aids for diagnostic interpretation have become part of medical practice [4, 5]. Similarly, in aviation, flight management systems control almost all parts of the flight [6, 7], and in surface transportation, public transportation is increasingly automated, and the first autonomous cars appear on public roads [8, 9]. Manuscript submitted October 30, 2018; (Corresponding author: Joachim Meyer) N. Douer with the Department of Industrial Engineering at Tel Aviv University, Ramat Aviv, Tel Aviv 69978, Israel (email: nirdouer@mail.tau.ac.il).