It's Time to Develop the Tools We Need to Hold Algorithms Accountable

When algorithms fail, people get hurt. We now have the evidence - of false arrests [1] and wrongful accusations [2] perpetuated by avoidable errors; glitches blocking access to healthcare [3] or housing [4]; and biased outcomes creating rather than removing barriers for the most vulnerable to succeed [5]. Much of this evidence was collected by algorithmic auditors, who meticulously analyze these systems for failures and communicate concretely about the ways in which these systems fall short of ensuring the safety of those impacted. Given the increasingly visible policy developments mandating audit activity and the proliferation of deployed algorithmic products, algorithmic audits are increasingly crucial tools for holding vendors and operators accountable for the impacts of the algorithmic systems they choose to release into the real world. However, despite the increasingly prevalent academic discussion of algorithmic audits, such audits remain incredibly difficult to execute. Audits are often completed with much difficulty and are surprisingly ad hoc, developed in isolation of other efforts and reliant on either custom tooling or mainstream resources that fall short of facilitating the actual audit goals of accountability.