Artificial Intelligence for National Security: The Predictability Problem

This report focuses on the risks related to the potential lack of predictability of AI systems – referred to as the predictability problem – and its implications for the governance of AI systems in the national security domain. Predictability of AI systems indicates the degree to which one can answer the question: what will an AI system do? The predictability problem can refer both to correct and incorrect outcomes of an AI system, as the issue is not whether the outcomes follow logically from the working of the system, but whether it is possible to foresee them at the time of deployment. In this report, we first analyse the predictability problem from technical and socio-technical perspectives and then focus on relevant UK, EU and US policy to consider whether and how they address this problem. From a technical perspective, we argue that given the multi- faceted process of design, development, and deployment of an AI system, it is not possible to account for all sources of errors or emerging behaviours that could result.