What are radiological deep learning models actually learning?

In radiology, we'd like deep learning models to identify patterns in imaging that suggest disease. For example, to detect pneumonia (lung infection), we'd like them to identify patterns in the lung that indicate the presence of an active infection. But do we know that is what they're actually doing? My collaborators and I recently released a preprint on arXiv examining how confounding variables may degrade the generalization performance of a CNN trained to identify pneumonia. Let me take a step back and give some examples of the problem that motivates this work.