A clinical artificial intelligence (AI) system is often validated on a held-out set of data which it has not been exposed to before (e.g., data from a different hospital with a distinct electronic health record system). This evaluation process is meant to mimic the deployment of an AI system on data in the wild; those which are currently unseen by the system yet are expected to be encountered in a clinical setting. However, when data in the wild differ from the held-out set of data, a phenomenon referred to as distribution shift, and lack ground-truth annotations, it becomes unclear the extent to which AI-based findings can be trusted on data in the wild. Here, we introduce SUDO, a framework for evaluating AI systems without ground-truth annotations. SUDO assigns temporary labels to data points in the wild and directly uses them to train distinct models, with the highest performing model indicative of the most likely label. Through experiments with AI systems developed for dermatology images, histopathology patches, and clinical reports, we show that SUDO can be a reliable proxy for model performance and thus identify unreliable predictions. We also demonstrate that SUDO informs the selection of models and allows for the previously out-of-reach assessment of algorithmic bias for data in the wild without ground-truth annotations. The ability to triage unreliable predictions for further inspection and assess the algorithmic bias of AI systems can improve the integrity of research findings and contribute to the deployment of ethical AI systems in medicine.

Now you can monitor your deep learning model using an application called'TensorDash' remotely instead of sitting in front of your workstation to monitor your DL model's progress. 'TensorDash' lets you remotely monitor your deep learning model's metrics and notifies you when your model training is completed or crashed. This could be really helpful while training a model on the cloud (Google Colab). Note: These installation steps assume that you are on a Linux or Mac environment. If you are on Windows, you will need to remove sudo to run the commands below.

As described in the Part 1, I wanted to deploy my Deep Learning model into production. I've shown how to prepare the model for TensorFlow Serving. We exported the GAN model as Protobuf and it is now ready to be hosted. TensorFlow Serving implements a server that processes incoming requests and forwards them to a model. This server could be running somewhere, most probably, at your Cloud provider (such as Amazon AWS, Google Cloud Platform, Microsoft Azure), to be available to the world.