Artificial-intelligence algorithms are processing vast troves of data in electronic medical records, searching for patterns to predict future outcomes and recommend treatments. They are creating early-warning systems to help hospital staff spot subtle but serious changes in a patient's condition that aren't always visible or noticed in a busy unit, and predicting which patients about to be discharged from the hospital are at highest risk of being readmitted. These systems are just one effort in a vast array of AI projects in healthcare--from helping detect cancer in radiology images to identifying which drugs to test on patients with different diseases. But this prediction technology holds especially significant promise to transform care and improve patient safety in ER and ICU cases--as long as the systems can be designed to avoid some of the medical, technological and ethical concerns that have emerged in mixing the science of machine learning with the art of medicine. Sepsis Watch is based on data from 42,000 inpatient encounters, with 21.3% of those patients having experienced sepsis.

Artificial-intelligence algorithms are processing vast troves of data in electronic medical records, searching for patterns to predict future outcomes and recommend treatments. They are creating early-warning systems to help hospital staff spot subtle but serious changes in a patient's condition that aren't always visible or noticed in a busy unit, and predicting which patients about to be discharged from the hospital are at highest risk of being readmitted. These systems are just one effort in a vast array of AI projects in healthcare--from helping detect cancer in radiology images to identifying which drugs to test on patients with different diseases. But this prediction technology holds especially significant promise to transform care and improve patient safety in ER and ICU cases--as long as the systems can be designed to avoid some of the medical, technological and ethical concerns that have emerged in mixing the science of machine learning with the art of medicine. Sepsis Watch is based on data from 42,000 inpatient encounters, with 21.3% of those patients having experienced sepsis.

Artificial intelligence algorithms are everywhere in healthcare. They sort through patients' data to predict who will develop medical conditions like heart disease or diabetes, they help doctors figure out which people in an emergency room are the sickest, and they screen medical images to find evidence of diseases. But even as AI algorithms become more important to medicine, they're often invisible to people receiving care. Artificial intelligence tools are complicated computer programs that suck in vast amounts of data, search for patterns or trajectories, and make a prediction or recommendation to help guide a decision. Sometimes, the way algorithms process all of the information they're taking in is a black box -- inscrutable even to the people who designed the program.

Artificial intelligence is often framed in terms of headline-grabbing technology and dazzling promise. But some of the workers who enable these programs -- the people who do things like code data, flag pictures, or work to integrate the programs into the workplace -- are often overlooked or undervalued. "This is a common pattern in the social studies of technology," said Madeleine Clare Elish, SM '10, a senior research scientist at Google. "A focus on new technology, the latest innovation, comes at the expense of the humans who are working to actually allow that innovation to function in the real world." Speaking at the recent EmTech Digital conference hosted by MIT Technology Review, Elish and other researchers said artificial intelligence programs often fail to account for the humans who incorporate AI systems into existing workflow, workers doing behind-the-scenes labor to make the programs run, and the people who are negatively affected by AI outcomes.

In November of 2018, a new deep-learning tool went online in the emergency department of the Duke University Health System. Called Sepsis Watch, it was designed to help doctors spot early signs of one of the leading causes of hospital deaths globally. Sepsis occurs when an infection triggers full-body inflammation and ultimately causes organs to shut down. It can be treated if diagnosed early enough, but that's a notoriously hard task because its symptoms are easily mistaken for signs of something else. Sepsis Watch promised to change that.

Nurse Dina Sarro didn't know much about artificial intelligence when Duke University Hospital installed machine learning software to raise an alarm when a person was at risk of developing sepsis, a complication of infection that is the number one killer in US hospitals. The software, called Sepsis Watch, passed alerts from an algorithm Duke researchers had tuned with 32 million data points from past patients to the hospital's team of rapid response nurses, co-led by Sarro. But when nurses relayed those warnings to doctors, they sometimes encountered indifference or even suspicion. When docs questioned why the AI thought a patient needed extra attention, Sarro found herself in a tough spot. "I wouldn't have a good answer because it's based on an algorithm," she says.

Responsible for more than 270,000 annual deaths in the U.S., sepsis claims a life in this country every two minutes. The condition, which arises from the body's inflammatory response to infection, costs over $27 billion in hospitalizations each year. Despite advancements in understanding and managing sepsis, the fight is far from over. This is why an evolved strategy using predictive technology is critical. By leveraging patient data, artificial intelligence is helping healthcare organizations identify patients in the early stages of sepsis.

Machine learning technologies are increasingly developed for use in healthcare. While research communities have focused on creating state-of-the-art models, there has been less focus on real world implementation and the associated challenges to accuracy, fairness, accountability, and transparency that come from actual, situated use. Serious questions remain under examined regarding how to ethically build models, interpret and explain model output, recognize and account for biases, and minimize disruptions to professional expertise and work cultures. We address this gap in the literature and provide a detailed case study covering the development, implementation, and evaluation of Sepsis Watch, a machine learning-driven tool that assists hospital clinicians in the early diagnosis and treatment of sepsis. We, the team that developed and evaluated the tool, discuss our conceptualization of the tool not as a model deployed in the world but instead as a socio-technical system requiring integration into existing social and professional contexts. Rather than focusing on model interpretability to ensure a fair and accountable machine learning, we point toward four key values and practices that should be considered when developing machine learning to support clinical decision-making: rigorously define the problem in context, build relationships with stakeholders, respect professional discretion, and create ongoing feedback loops with stakeholders. Our work has significant implications for future research regarding mechanisms of institutional accountability and considerations for designing machine learning systems. Our work underscores the limits of model interpretability as a solution to ensure transparency, accuracy, and accountability in practice. Instead, our work demonstrates other means and goals to achieve FATML values in design and in practice.

In hospitals, doctors and nurses keep vigilant watch over patients' vital signs and blood tests to catch the first symptoms of sepsis. In this life-threatening condition, the body responds to an infection with widespread inflammation that can lead to organ failure. Cases can progress rapidly to severe sepsis and then to septic shock, which has a mortality rate of almost 50 percent in the United States. But even the most vigilant humans get tired, make mistakes, and miss subtle patterns. That's why several hospitals are experimenting with artificially intelligent sepsis detectors.