When searching for talent, sometimes the best person for the job is a machine. Robots make sense for repetitive and dangerous tasks, but they also work well as a check against bias. Artificial intelligence already outperforms judges in choices about setting bail because humans on the bench tend to overthink the defendants' demeanor, a poor predictor of flight risk. Likewise, hiring algorithms do better than recruiters at screening resumes because humans in HR show too much favoritism for traditional applicants. Unfortunately, smart technology also has blind spots.

There is a growing demand to be able to “explain” machine learning (ML) systems' decisions and actions to human users, particularly when used in contexts where decisions have substantial implications for those affected and where there is a requirement for political accountability or legal compliance ([ 1 ][1]). Explainability is often discussed as a technical challenge in designing ML systems and decision procedures, to improve understanding of what is typically a “black box” phenomenon. But some of the most difficult challenges are nontechnical and raise questions about the broader accountability of organizations using ML in their decision-making. One reason for this is that many decisions by ML systems may exhibit bias, as systemic biases in society lead to biases in data used by the systems ([ 2 ][2]). But there is another reason, less widely appreciated. Because the quantities that ML systems seek to optimize have to be specified by their users, explainable ML will force policy-makers to be more explicit about their objectives, and thus about their values and political choices, exposing policy trade-offs that may have previously only been implicit and obscured. As the use of ML in policy spreads, there may have to be public debate that makes explicit the value judgments or weights to be used. Merely technical approaches to “explaining” ML will often only be effective if the systems are deployed by trustworthy and accountable organizations. The promise of ML is that it could lead to better decisions, yet concerns have been raised about its use in policy contexts such as criminal justice and policing. A fundamental element of the demand for explainability is for explanation of what the system is “trying to achieve.” Most policy decision-making makes extensive use of constructive ambiguity to pursue shared objectives with sufficient political consensus. There is thus a tension between political or policy decisions, which trade off multiple (often incommensurable) aims and interests, and ML, typically a utilitarian maximizer of what is ultimately a single quantity and which typically entails explicit weighting of decision criteria. We focus on public policy decision-making using ML algorithms that learn the relationships between data inputs and decision outputs. As a first step, policy-makers need to decide among a number of possible meanings of explainability. These range from causal accounts and post hoc interpretations of decisions ([ 3 ][3]) to assurance that outcomes are reliable or fair in terms of the specified objectives for the system ([ 4 ][4]). For example, the explainability requirements for ML systems used by local authorities to determine benefit payments will differ greatly from those required for the enforcement of competition policy with respect to pricing by online merchants. Each of the specific meanings of explainability has different technical requirements, which will imply choices about where efficiency and cost might need to be sacrificed to deliver both explainability and the desired outcomes. Choosing which meaning is relevant is far from a technical question (though what can be provided depends on what is technically feasible). Thus, those seeking explainability will need to specify, in terms translatable to how ML systems operate, what exactly they mean, and what kind of evidence would satisfy their demand ([ 5 ][5]). It must also be possible to monitor whatever explanations are provided, and there must be practical methods to enforce compliance. Furthermore, policy institutions starting to deploy algorithmic or ML-based decision systems, such as the police, courts, and government agencies, are operating in the context of declining trust in some aspects of public life. This context is important for understanding demands for explainability, as these may in part reflect broader legitimacy demands of the policy-making process. If an organization is not trusted, its automated decision procedures will likely also be distrusted. This implies a broader need for trustworthy processes and institutions, for “intelligent accountability” as the result of informed and independent scrutiny, communicated clearly to the public ([ 6 ][6]). Satisfying the demand for explainability implies testing the trustworthiness of the organizations using ML systems to make decisions affecting individuals. Evaluation requires comparing outcomes against a benchmark, which can be the baseline situation, or a specified desired outcome. Taking the demand for explainability as a demand for accountability, the promise of ML is that it could lead to more legitimate and better decisions than humans can make, on some measure. Potential benefits are clearly demonstrable in some forms of medical diagnosis ([ 7 ][7]) or monitoring attempted financial fraud ([ 8 ][8]). In these domains, there is general agreement on a straightforward quantity to optimize, and the incentives of principals (citizens or customers) and agents (public or corporate decision-makers) are aligned. Public concern about the use of ML focuses on other domains, such as marketing or policing, where there may be less agreement about (or trust in) the aim of either the ML system or the organization using it. These concerns highlight a key challenge posed by the use of ML in policy decisions, which is that ML processes are almost always set up to optimize an objective function; this optimization goal can be described in anthropomorphic terms as the “intention” of the system. Yet there is often little or no explicit discussion by policy-makers when considering using ML systems about what conflicting goals, benefits, and risks may trade off against each other as a result. One reason for this is that it is inherently challenging to specify a concrete objective function in sociopolitical domains ([ 9 ][9]). For example, like current ML systems, economists' decisions are informed by estimates of statistical relationships between directly observable and unobservable variables, derived from data generated by a complex environment. Yet economic policies such as tax changes often fail to take into account all relevant factors in the decision environment, or likely behavior changes, in specifying the objective function ([ 10 ][10]). The use of ML systems in other policy contexts will expand the scope of such unintended consequences. Given that the dominant paradigm of machine learning is based on optimization, the use of ML in policy decisions thus speaks to a fundamental debate about social welfare. From the perspective of ethical theories, ML is largely consequentialist: A machine system is configured on the basis of its ability to achieve a desired outcome. Conventional policy analysis is similarly typically based on consequentialist economic social welfare criteria. The well-known impossibility theorems in social choice theory ([ 11 ][11]) establish that when the goal is to aggregate individual choices under a set of reasonable social decision rules, it is impossible to satisfy a set of desirable criteria simultaneously, and thus impossible to achieve a set of desired outcomes by optimizing a single quantity. Critics of consequentialist economic policy analysis argue that people have multidimensional, probably incommensurable, and possibly contradictory objectives, so that imposing utilitarian decision-making procedures will conflict both with reality and with ethical intuitions ([ 12 ][12]). Nevertheless, policy choices are made, so there has always been an unavoidable, albeit often implicit, trade-off or weighting of different objectives ([ 12 ][12]). For example, cost-benefit analysis can incorporate environmental and cultural, as well as financial, considerations, but converts all of these into monetary values. Any choice made when there are multiple interests or trade-offs will imply weights on the different components. As these trade-offs are codified into ML objective functions, the weights given to competing objectives comprise a first-line characterization of how conflicts will be resolved. Using ML systems in political contexts is extending the use of optimization; progress in making these ML systems more understandable to policy-makers will make the de facto choices between competing objectives more explicit than they have been previously ([ 13 ][13]). Greater explainability is therefore likely to have to lead to a more explicit political, not wholly technical, debate. Distilling concrete, unambiguous objectives in this way may turn out to be extremely challenging, for ambiguity about objectives is often useful in policy-making precisely because it blurs uncomfortable conflicts of interest. In many domains, policies generally emerge as a pragmatic compromise between fundamentally conflicting aims. For example, people who disagree about whether the justice system should be retributive or rehabilitative may well be able to agree on specific sentencing policies. Such incompletely theorized agreements “Play an important function in any well-functioning democracy consisting of a heterogeneous population” ([ 14 ][14], p. 1738). The omission of discussion of ultimate aims can make it easier to achieve consensus on difficult issues. As there is some (limited) scope to interpret means to achieve the objective with flexibility, the “weighting” of different fundamental aims remains implicit, and diverse political communities can make progress. An optimistic conclusion would be that being forced by the use of ML systems to be more explicit about policy objectives could promote useful debate leading in the long run to more considered outcomes. ML systems can be used to explore choices and outcomes on different counterfactual high-level objectives, such as retribution or rehabilitation in justice, enabling considered human judgments. However, it may in practice be impossible to specify what we collectively truly want in rigid code. For example, many local governments do not seem to be engaging in public consultation when they adopt predictive ML systems, such as to flag “troubled” families that are likely to need interventions. Although steps such as explicitly adding uncertainty to the ML objective might address this challenge of imperfectly specified objectives in future, ML systems are unable at present to offer wisely moderated solutions to ambiguous objectives ([ 15 ][15]). Human decision-makers can make use of common sense or tacit knowledge, and often override decisions indicated by an economic model or other formal policy analysis, and they will be able to do the same when assisted by ML. Yet, demanding that ML systems be explainable is likely to make the trade-offs between different objectives far more explicit than has been the norm previously. Ultimately, the use of explainable ML systems in the public sector will make a broader debate about social objectives and social justice newly salient. Providing explanations requires being transparent about the systems' objectives — forcing clarity about choices and trade-offs previously often made implicitly — and how their predictions or decisions draw on patterns revealed by a fundamentally biased social and institutional system. Moreover, whereas democratic political systems often look to resolve conflicts through constructive ambiguity—or in other words, the failure to explain—ML systems may require ambiguous objectives to be resolved unequivocally. So, although the need for explainability certainly poses technical challenges, it poses political challenges too, which have not to date been widely acknowledged. Yet, the increasing scope of ML, and progress in delivering explainability, in politically salient areas of policy could shine a helpful spotlight on the conflicting aims and the implicit trade-offs in policy decisions, just as it already has on the biases in existing social and economic systems. 1. [↵][16]1. B. Dattner, 2. T. Chamorro-Premuzic, 3. R. Buchband, 4. L. Schittler , The legal and ethical implications of using AI in hiring, Harv. Bus. Rev. April, 25 (2019); . 2. [↵][17]1. R. Richardson, 2. J. Schultz, 3. K. Crawford , New York Univ. Law Rev. 192, 204 (2019). [OpenUrl][18] 3. [↵][19]1. Z. Lipton , The mythos of model interpretability (2017); . 4. [↵][20]1. T. Miller , Explanation in artificial intelligence: Insights from the social sciences (2018); . 5. [↵][21]1. P. Madumal, 2. T. Miller, 3. L. Sonenberg, 4. F. Vetere , A grounded interaction protocol for explainable artificial intelligence (2019); . 6. [↵][22]1. O. O'Neill , Int. J. Philos. Stud. 26, 293 (2018). [OpenUrl][23] 7. [↵][24]1. J. De Fauw et al ., Nat. Med. 24, 1342 (2018). [OpenUrl][25][PubMed][26] 8. [↵][27]1. T. Lynn, 2. G. Mooney, 3. P. Rosati, 4. M. Cummins 1. S. Aziz, 2. M. Dowling , in Disrupting Finance: FinTech and Strategy in the 21st Century, T. Lynn, G. Mooney, P. Rosati, M. Cummins, Eds. (Palgrave, 2019), pp. 33–50. 9. [↵][28]1. P. Samuelson , Foundations of Economic Analysis (Harvard University Press, 1979), chap. 8, pp. 203–252. 10. [↵][29]1. J. Le Grand , Br. J. Polit. Sci. 21, 423 (1991). [OpenUrl][30][CrossRef][31][Web of Science][32] 11. [↵][33]1. A. Sen , Am. Econ. Rev. 89, 349 (1999). [OpenUrl][34][CrossRef][35][Web of Science][36] 12. [↵][37]1. E. Anderson , Value in Ethics and Economics (Harvard Univ. Press, 1993). 13. [↵][38]1. S. Grover, 2. C. Pulice, 3. G. I. Simari, 4. V. S. Subrahmanian , IEEE Trans. Comput. Soc. Syst. 6, 350 (2019). [OpenUrl][39] 14. [↵][40]1. C. R. Sunstein , Harv. Law Rev. 108, 1733 (1995). [OpenUrl][41][CrossRef][42][Web of Science][43] 15. [↵][44]1. M. Hildebrandt , Smart Technologies and the End(s) of Law (Edward Elgar, 2016). Acknowledgments: We are grateful to M. Kenny and N. Rabinowitz for helpful comments. A.W. acknowledges support from the David MacKay Newton research fellowship at Darwin College, The Alan Turing Institute under EPSRC grants EP/N510129/1 and TU/B/000074, the Leverhulme Trust via CFI, and the Centre for Data Ethics and Innovation. [1]: #ref-1 [2]: #ref-2 [3]: #ref-3 [4]: #ref-4 [5]: #ref-5 [6]: #ref-6 [7]: #ref-7 [8]: #ref-8 [9]: #ref-9 [10]: #ref-10 [11]: #ref-11 [12]: #ref-12 [13]: #ref-13 [14]: #ref-14 [15]: #ref-15 [16]: #xref-ref-1-1 "View reference 1 in text" [17]: #xref-ref-2-1 "View reference 2 in text" [18]: {openurl}?query=rft.jtitle%253DNew%2BYork%2BUniv.%2BLaw%2BRev.%26rft.volume%253D192%26rft.spage%253D204%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [19]: #xref-ref-3-1 "View reference 3 in text" [20]: #xref-ref-4-1 "View reference 4 in text" [21]: #xref-ref-5-1 "View reference 5 in text" [22]: #xref-ref-6-1 "View reference 6 in text" [23]: {openurl}?query=rft.jtitle%253DInt.%2BJ.%2BPhilos.%2BStud.%26rft.volume%253D26%26rft.spage%253D293%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [24]: #xref-ref-7-1 "View reference 7 in text" [25]: {openurl}?query=rft.jtitle%253DNat.%2BMed.%26rft.volume%253D24%26rft.spage%253D1342%26rft_id%253Dinfo%253Apmid%252Fhttp%253A%252F%252Fwww.n%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [26]: /lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fsci%2F368%2F6498%2F1433.atom [27]: #xref-ref-8-1 "View reference 8 in text" [28]: #xref-ref-9-1 "View reference 9 in text" [29]: #xref-ref-10-1 "View reference 10 in text" [30]: {openurl}?query=rft.jtitle%253DBr.%2BJ.%2BPolit.%2BSci.%26rft.volume%253D21%26rft.spage%253D423%26rft_id%253Dinfo%253Adoi%252F10.1017%252FS0007123400006244%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [31]: /lookup/external-ref?access_num=10.1017/S0007123400006244&link_type=DOI [32]: /lookup/external-ref?access_num=A1991GK10800002&link_type=ISI [33]: #xref-ref-11-1 "View reference 11 in text" [34]: {openurl}?query=rft.jtitle%253DAm.%2BEcon.%2BRev.%26rft.volume%253D89%26rft.spage%253D349%26rft_id%253Dinfo%253Adoi%252F10.1257%252Faer.89.3.349%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [35]: /lookup/external-ref?access_num=10.1257/aer.89.3.349&link_type=DOI [36]: /lookup/external-ref?access_num=000081083000001&link_type=ISI [37]: #xref-ref-12-1 "View reference 12 in text" [38]: #xref-ref-13-1 "View reference 13 in text" [39]: {openurl}?query=rft.jtitle%253DIEEE%2BTrans.%2BComput.%2BSoc.%2BSyst.%26rft.volume%253D6%26rft.spage%253D350%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [40]: #xref-ref-14-1 "View reference 14 in text" [41]: {openurl}?query=rft.jtitle%253DHarv.%2BLaw%2BRev.%26rft.volume%253D108%26rft.spage%253D1733%26rft_id%253Dinfo%253Adoi%252F10.2307%252F1341816%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [42]: /lookup/external-ref?access_num=10.2307/1341816&link_type=DOI [43]: /lookup/external-ref?access_num=A1995RA49100010&link_type=ISI [44]: #xref-ref-15-1 "View reference 15 in text"

In 2015, alarmed by an escalating civil war in Yemen, Saudi Arabia led an air campaign against the country to defeat what it deemed a threatening rise of Shia power. The intervention, launched with eight other largely Sunni Arab states, was meant to last only a few weeks, Saudi officials had said. Nearly five years later, it still hasn't stopped. By some estimates, the coalition has since carried out over 20,000 air strikes, many of which have killed Yemeni civilians and destroyed their property, allegedly in direct violation of international law. Human rights organizations have since sought to document such war crimes in an effort to stop them through legal challenges.

Artificial intelligence and machine learning technologies are poised to supercharge productivity in the knowledge economy, transforming the future of work. Machine learning (ML)--technology in which algorithms "learn" from existing patterns in data to conduct statistically driven predictions and facilitate decisions--has been found in multiple contexts to reveal bias. Such biases often result from slanted training data or skewed algorithms. It comes when outside individuals stand to benefit from bias predictions, and work to strategically alter the inputs. A couple of the most common contexts are perhaps job applicants and people making a claim against their insurance.

A high-level goal of many AI projects is to address the ethical implications of algorithms along the lines of fairness and discrimination. It is a known fact that algorithms can facilitate illegal discrimination. For example, it may not surprise that each investor wants to put more capital in loans with a high return of investment and low risk. A modern idea is to use a machine learning model to decide, based on the sliver of known information about the outcome of past loans, which future loan requests give the largest chance of the borrower fully paying it back while achieving the best trade-off with high returns (high-interest rate). There's one problem: the model is trained on historical data, and poor uneducated people, often racial minorities or people with less working experience have a historical trend of being more likely to succumb to loan charge-off than the general population.

Boston has become the second-largest city in the US to bar police and other local agencies from using facial recognition software (via WBUR). The city's 13-member council voted unanimously to ban the technology outside of use in specific criminal cases. The bill also prevents any city official from obtaining the technology through a third-party. The bill was put forward in part to protect the city's minority residents. "Boston should not use racially discriminatory technology that threatens the privacy and basic rights of our residents," said councilor Michelle Wu, who co-sponsored the bill alongside councilor Ricardo Arroyo.

Talking about racism and it's impact on robotics and roboticists was the first conversation in our new biweekly online discussion series "Society, Robots and Us" on alternate Tuesdays at 6pm PDT. It was a generous, honest and painful discussion that I hope has left a lasting impact on everyone who listened. There is systemic racism in America, and this does have an impact on robotics and roboticists in many many ways. The US Senator Elizabeth Warren in conversation today with Alicia Garza from Black Futures Lab said, "America was founded on principles of liberty and freedom, but it was built on the backs of enslaved people. This is a truth we must not ignore. Racism and white supremacy have shaped every crucial aspect of our economy, and our political system for generations now."

Boston City Councilors voted unanimously to ban the use of facial-recognition technology by police -- technology the Boston Police Department currently doesn't use anyway due to its unreliability. All 13 councilors voted in favor of the order authored by Councilors Ricardo Arroyo and Michelle Wu to ban the city from using technology that matches people's faces. Mayor Marty Walsh's office said the mayor would review the legislation, not committing to whether he'd sign it or not. "It puts Bostonians at risk for misidentification," Arroyo said. A recent MIT study found that the technology was wrong more often when trying to identify darker-skinned people.

A US university's claim it can use facial recognition to "predict criminality" has renewed debate over racial bias in technology. Harrisburg University researchers said their software "can predict if someone is a criminal, based solely on a picture of their face". The software "is intended to help law enforcement prevent crime", it said. But 1,700 academics have signed an open letter demanding the research remains unpublished. One Harrisburg research member, a former police officer, wrote: "Identifying the criminality of [a] person from their facial image will enable a significant advantage for law-enforcement agencies and other intelligence agencies to prevent crime from occurring."

These past few months have been especially challenging, and the deployment of technology in ways hitherto untested at an unrivalled pace has left the internet and technology watchers aghast. Artificial intelligence has become the byword for technological progress and is being used in everything from helping us combat the COVID-19 pandemic to nudging our attention in different directions as we all spend increasingly larger amounts of time online. It has never been more important that we keep a sharp eye out on the development of this field and how it is shaping our society and interactions with each other. With this inaugural edition of the State of AI Ethics we hope to bring forward the most important developments that caught our attention at the Montreal AI Ethics Institute this past quarter. Our goal is to help you navigate this ever-evolving field swiftly and allow you and your organization to make informed decisions. This pulse-check for the state of discourse, research, and development is geared towards researchers and practitioners alike who are making decisions on behalf of their organizations in considering the societal impacts of AI-enabled solutions. We cover a wide set of areas in this report spanning Agency and Responsibility, Security and Risk, Disinformation, Jobs and Labor, the Future of AI Ethics, and more. Our staff has worked tirelessly over the past quarter surfacing signal from the noise so that you are equipped with the right tools and knowledge to confidently tread this complex yet consequential domain.