John Quackenbush was frustrated with Google. It was January 2020, and a team led by researchers from Google Health had just published a study in Nature about an artificial intelligence (AI) system they had developed to analyze mammograms for signs of breast cancer. The system didn't just work, according to the study, it worked exceptionally well. When the team fed it two large sets of images to analyze--one from the UK and one from the U.S.--it reduced false positives by 1.2 and 5.7 percent and false negatives by 2.7 and 9.4 percent compared with the original determinations made by medical professionals. In a separate test that pitted the AI system against six board-certified radiologists in analyzing nearly 500 mammograms, the algorithm outperformed each of the specialists. The authors concluded that the system was "capable of surpassing human experts in breast cancer prediction" and ready for clinical trials. An avalanche of buzzy headlines soon followed. "Google AI system can beat doctors at detecting breast cancer," a CNN story declared.

Over the course of his career, Quackenbush has been a vocal advocate for transparency and data sharing, so much so that President Barack Obama named him a White House Open Science Champion of Change in 2013 for his efforts to ensure that vast amounts of genomic data are accessible to researchers around the world. Reproducibility is the essence of the scientific method, Quackenbush says, and it is of the utmost importance when new technologies are being floated for use in human clinical trials. As a cautionary tale, Quackenbush mentions Anil Potti, a former Duke University professor who in the early 2000s claimed to have discovered genetic signatures that could determine how individuals with certain cancers would respond to chemotherapy. The technique sailed into human clinical trials even though other researchers reported that they were unable to reproduce Potti's findings. Eventually it was revealed that Potti had falsified data and study results, and the whole house of biomedical cards came crashing down.

Boston, MA – Scientists working at the intersection of Artificial Intelligence (AI) and cancer care need to be more transparent about their methods and publish research that is reproducible, according to a new commentary co-authored by John Quackenbush, Henry Pickering Walcott Professor of Computational Biology and Bioinformatics and chair of the Department of Biostatistics at Harvard T.H. Chan School of Public Health. "The foundation of the scientific method is that research results must be testable by others. Testability is even more important in clinical applications because we need a high level of confidence in our methods before they are used with patients," Quackenbush said. "In applications of Artificial Intelligence, this requires that the models, software code, and data are available for independent validation. Transparency will accelerate research, advance patient care, and will build confidence among scientists and clinicians."

In a rush to find solutions for the Covid-19 pandemic, researchers are deploying machine learning algorithms to trawl through data that might give us more clues about the virus. Some claim to have identified potential treatments based on the data, while others are using it to screen patients or identify those at highest risk. But, like their vaccine and drug counterparts, many of these algorithms are still unproven. With hundreds of research articles describing the use of artificial intelligence or machine learning -- many of them preprints -- it can be difficult to sort out which ones are most effective. "I've heard a lot of hype about machine learning being applied to battling Covid-19, but I haven't seen very many concrete examples where you could imagine in the short- or medium-term something that is going to have a substantial effect," said John Quackenbush, chair of the Department of Biostatistics at the Harvard T.H. Chan School of Public Health, in a phone interview.

When it comes to data curation, the problem isn't the rise of Big Data, but the haphazard way data often present themselves. That's how John Quackenbush characterized the issue in a panel Wednesday morning at the MedCity CONVERGE conference in Philadelphia on practical applications of artificial intelligence (AI) and machine learning (ML) in oncology. Quackenbush, the director of the Center for Cancer Computational Biology at the Dana-Farber Cancer Institute in Boston, was referring to the difficulties of faced by curators when they had to go into clinical trial protocol pages and take down the studies' entry criteria manually due to the inconsistent way they were written. "I like to characterize it not as a Big Data problem, but as a messy data problem," he said. Moderator Ayan Bhattacharya, who serves as advanced analytics specialist leader at Deloitte Consulting, noted that health management organizations, health plans and others have been investing in technology to assist curation that had previously been the work of human editors.