A multiple testing framework for diagnostic accuracy studies with co-primary endpoints

This is indicated, among others, by several review and overview publications (Ching et al., 2018; Jiang et al., 2017; Litjens et al., 2017; Miotto, Wang, Wang, Jiang, & Dudley, 2017). In particular, the capabilities of end-to-end deep learning approaches on such supervised learning tasks are highly promising. For instance, vast advances have been reported in the literature regarding cancer diagnosis with deep neural networks (Hu et al., 2018). End-to-end deep learning refers to a trend involving deep (neural network) model architectures which are able to learn highly complex relationships between predictors and the target variable while having less parameters than traditional (more shallow) models with comparable performance (Goodfellow, Bengio, & Courville, 2016). In the training process, highly complex features are derived automatically by the learning algorithm (LeCun, Bengio, & Hinton, 2015). This framework contrasts the traditional pipeline of domain specific data preprocessing and handcrafted features in combination with simpler prediction models. Despite all the recent success of machine learning, there are still challenges regarding over-optimistic conclusions drawn from finite datasets which may to a large extend be attributed to the following two (broad) categories: 1. Study design and reporting: The most popular recommendation to split data for training, selection and evaluation is frequently employed in practice (Friedman, Hastie, & Tibshirani, 2009; Géron, 2017; Goodfellow et al., 2016; Japkowicz & Shah, 2011; Kuhn & Johnson, 2013; Zheng, 2015). In the ML community, the according datasets are commonly denoted as training, validation and test set.