Detecting Memorization in ReLU Networks

arXiv.org Machine Learning

We propose a new notion of'non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the nonnegative rank of the activation matrix. Considering batches of similar samples, we find that high non-linearity in deep layers is indicative of memorization. Furthermore, by applying our approach layer-by-layer, we find that the mechanism for memorization consists of distinct phases. We perform experiments on fully-connected and convolutional neural networks trained on several image and audio datasets. Our results demonstrate that as an indicator for memorization, our technique can be used to perform early stopping. A fundamental challenge in machine learning is balancing the bias-variance tradeoff, where overly simple learning models underfit the data (suboptimal performance on the training data) and overly complex models are expected to overfit or memorize the data (perfect training set performance, but suboptimal test set performance). The latter direction of this tradeoff has come into question with the observation that deep neural networks do not memorize their training data despite having sufficient capacity to do so (Zhang et al., 2016), the explanation of which is a matter of much interest.


AI can speed up precision medicine, New York Genome Center-IBM Watson study shows

#artificialintelligence

The potential for artificial intelligence in precision medicine is big, according to conclusions of a new study by the New York Genome Center and IBM. The results, published in the July 11 issue of Neurology Genetics, a journal of the American Academy of Neurology, showed that researchers at the New York Genome Center, Rockefeller University and other institutions – along with IBM – verified the potential of IBM Watson for Genomics to analyze complex genomic data from state-of-the-art DNA sequencing of whole genomes. "This study documents the strong potential of Watson for Genomics to help clinicians scale precision oncology more broadly," Vanessa Michelini, Watson for Genomics Innovation Leader for IBM Watson Health, said in a statement. "Clinical and research leaders in cancer genomics are making tremendous progress towards bringing precision medicine to cancer patients, but genomic data interpretation is a significant obstacle, and that's where Watson can help." The proof of concept study compared multiple techniques used to analyze genomic data from a glioblastoma patient's tumor cells and normal healthy cells, putting to work a beta version of Watson for Genomics technology to help interpret whole genome sequencing data for one patient.


UVA Scientists Use Machine Learning to Improve Gut Disease Diagnosis

#artificialintelligence

Machines use Google-type algorithms on biopsy images to help children get treatment faster. A study published in the open access journal JAMA Open Network today by scientists at the University of Virginia schools of Engineering and Medicine says machine learning algorithms applied to biopsy images can shorten the time for diagnosing and treating a gut disease that often causes permanent physical and cognitive damage in children from impoverished areas. In places where sanitation, potable water and food are scarce, there are high rates of children suffering from environmental enteric dysfunction, a disease that limits the gut's ability to absorb essential nutrients and can lead to stunted growth, impaired brain development and even death. The disease affects 20 percent of children under the age of 5 in low- and middle-income countries, such as Bangladesh, Zambia and Pakistan, but it also affects some children in rural Virginia. For Dr. Sana Syed, an assistant professor of pediatrics in the UVA School of Medicine, this project is an example of why she got into medicine.


How IBM Watson Overpromised and Underdelivered on AI Health Care

IEEE Spectrum Robotics

In 2014, IBM opened swanky new headquarters for its artificial intelligence division, known as IBM Watson. Inside the glassy tower in lower Manhattan, IBMers can bring prospective clients and visiting journalists into the "immersion room," which resembles a miniature planetarium. There, in the darkened space, visitors sit on swiveling stools while fancy graphics flash around the curved screens covering the walls. It's the closest you can get, IBMers sometimes say, to being inside Watson's electronic brain. One dazzling 2014 demonstration of Watson's brainpower showed off its potential to transform medicine using AI--a goal that IBM CEO Virginia Rometty often calls the company's moon shot. In the demo, Watson took a bizarre collection of patient symptoms and came up with a list of possible diagnoses, each annotated with Watson's confidence level and links to supporting medical literature. Within the comfortable confines of the dome, Watson never failed to impress: Its memory banks held knowledge of every rare disease, and its processors weren't susceptible to the kind of cognitive bias that can throw off doctors. It could crack a tough case in mere seconds. If Watson could bring that instant expertise to hospitals and clinics all around the world, it seemed possible that the AI could reduce diagnosis errors, optimize treatments, and even alleviate doctor shortages--not by replacing doctors but by helping them do their jobs faster and better.


Here's how often IBM's Watson agrees with doctors on the best way to treat cancer

#artificialintelligence

We're starting to get a better picture of how artificial intelligence could help doctors better treat cancer. And in data presented at the American Society of Clinical Oncology meeting, IBM Watson Health gave a snapshot of how it's playing out so far. The studies looked at concordance rates, or how often Watson for Oncology reached the same course of treatment as the cancer doctors at different cancer centers around the world. At Manipal Comprehensive Cancer Center in India, for 112 cases of lung cancer, there was 96.4% concordance between Watson and the doctors. For 126 cases of colon cancer it was 81% of the time, and for 124 cases of rectal cancer cases were 92.7%.