On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra-observer Variability in 2D Echocardiography Quality Assessment

--Uncertainty of labels in clinical data resulting from intra-observer variability can have direct impact on the reliability of assessments made by deep neural networks. In this paper, we propose a method for modelling such uncertainty in the context of 2D echocardiography (echo), which is a routine procedure for detecting cardiovascular disease at point-of-care. Echo imaging quality and acquisition time is highly dependent on the operator's experience level. Recent developments have shown the possibility of automating echo image quality quantification by mapping an expert's assessment of quality to the echo image via deep learning techniques. Nevertheless, the observer variability in the expert's assessment can impact the quality quantification accuracy. Here, we aim to model the intra-observer variability in echo quality assessment as an aleatoric uncertainty modelling regression problem with the introduction of a novel method that handles the regression problem with categorical labels. A key feature of our design is that only a single forward pass is sufficient to estimate the level of uncertainty for the network output. Compared to the 0 .11 The simplicity of the proposed approach means that it could be generalized to other applications of deep learning in medical imaging, where there is often uncertainty in clinical labels. Z. Liao and H. Girgis have contributed equally to this work. Abolmaesumi have contributed equally to the manuscript (emails: t.tsang@ubc.ca, Z. Liao, A. Abdi, H. V aseli, and J. Hetherington are with the Department of Electrical and Computer Engineering, The University of British Columbia, V ancouver, BC V6T 1Z4, Canada. H. Girgis, T. Tsang, and K. Gin are with V ancouver General Hospital Echocardiography Laboratory, Division of Cardiology, Department of Medicine, The University of British Columbia, V ancouver, BC V5Z 1M9, Canada. R. Rohling is with the Department of Electrical and Computer Engineering and the Department of Mechanical Engineering, The University of British Columbia, V ancouver, BC V6T 1Z4, Canada T. Tsang is the Director of the V ancouver General Hospital and University of British Columbia Echocardiography Laboratories, and Principal Investigator of the CIHR-NSERC grant supporting this work. Abolmaesumi is Co-Principal Investigator for the grant supporting this work and is with the Department of Electrical and Computer Engineering, The University of British Columbia, V ancouver, BC V6T 1Z4, Canada.