Combining Augmented Reality with Deep Learning for Cancer Diagnostics

Right: A picture of the prototype which has been retrofitted into a typical clinical-grade light microscope. Applications of deep learning in medical disciplines including ophthalmology, dermatology, radiology and pathology have recently shown great promise to increase both the accuracy and availability of high-quality healthcare. To further this technology, Google researchers have developed a tool that combines augmented reality with a deep learning neural network to provide pathologists with help in spotting cancerous cells on slides under a microscope. The prototype Augmented Reality Microscope (ARM) platform consists of a modified light microscope that enables real-time image analysis and presentation of the results of machine learning algorithms directly into the field of view. The ARM can be retrofitted into existing light microscopes found in hospitals and clinics by using low-cost, readily available components, and without the need for whole slide digital versions of the tissue being analyzed.