Jersakova, Radka
A large-scale and PCR-referenced vocal audio dataset for COVID-19
Budd, Jobie, Baker, Kieran, Karoune, Emma, Coppock, Harry, Patel, Selina, Cañadas, Ana Tendero, Titcomb, Alexander, Payne, Richard, Hurley, David, Egglestone, Sabrina, Butler, Lorraine, Mellor, Jonathon, Nicholson, George, Kiskin, Ivan, Koutra, Vasiliki, Jersakova, Radka, McKendry, Rachel A., Diggle, Peter, Richardson, Sylvia, Schuller, Björn W., Gilmour, Steven, Pigoli, Davide, Roberts, Stephen, Packham, Josef, Thornley, Tracey, Holmes, Chris
The UK COVID-19 Vocal Audio Dataset is designed for the training and evaluation of machine learning models that classify SARS-CoV-2 infection status or associated respiratory symptoms using vocal audio. The UK Health Security Agency recruited voluntary participants through the national Test and Trace programme and the REACT-1 survey in England from March 2021 to March 2022, during dominant transmission of the Alpha and Delta SARS-CoV-2 variants and some Omicron variant sublineages. Audio recordings of volitional coughs, exhalations, and speech were collected in the 'Speak up to help beat coronavirus' digital survey alongside demographic, self-reported symptom and respiratory condition data, and linked to SARS-CoV-2 test results. The UK COVID-19 Vocal Audio Dataset represents the largest collection of SARS-CoV-2 PCR-referenced audio recordings to date. PCR results were linked to 70,794 of 72,999 participants and 24,155 of 25,776 positive cases. Respiratory symptoms were reported by 45.62% of participants. This dataset has additional potential uses for bioacoustics research, with 11.30% participants reporting asthma, and 27.20% with linked influenza PCR test results.
Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers
Coppock, Harry, Nicholson, George, Kiskin, Ivan, Koutra, Vasiliki, Baker, Kieran, Budd, Jobie, Payne, Richard, Karoune, Emma, Hurley, David, Titcomb, Alexander, Egglestone, Sabrina, Cañadas, Ana Tendero, Butler, Lorraine, Jersakova, Radka, Mellor, Jonathon, Patel, Selina, Thornley, Tracey, Diggle, Peter, Richardson, Sylvia, Packham, Josef, Schuller, Björn W., Pigoli, Davide, Gilmour, Steven, Roberts, Stephen, Holmes, Chris
Recent work has reported that respiratory audio-trained AI classifiers can accurately predict SARS-CoV-2 infection status. Here, we undertake a large-scale study of audio-based AI classifiers, as part of the UK government's pandemic response. We collect a dataset of audio recordings from 67,842 individuals, with linked metadata, of whom 23,514 had positive PCR tests for SARS-CoV-2. In an unadjusted analysis, similar to that in previous works, AI classifiers predict SARS-CoV-2 infection status with high accuracy (ROC-AUC=0.846 However, after matching on measured confounders, such as selfreported symptoms, performance is much weaker (ROC-AUC=0.619 Upon quantifying the utility of audio-based classifiers in practical settings, we find them to be outperformed by predictions based on user-reported symptoms. We make best-practice recommendations for handling recruitment bias, and for assessing audio-based classifiers by their utility in relevant practical settings. Our work provides novel insights into the value of AI audio analysis and the importance of study design and treatment of confounders in AI-enabled diagnostics. The coronavirus disease 2019 (COVID-19) pandemic has been estimated by the World Health Organization (WHO) to have caused 14.9 million excess deaths over the 2020-2021 period (link). Table S1 summarises nine highly cited datasets and corresponding classification performance. Here, we analyse the largest PCR-validated dataset collected to date in the field of audio-based COVID-19 screening (ABCS). We design and specify an analysis plan in advance, to investigate whether using audio-based classifiers can improve the accuracy of COVID-19 screening over using self-reported symptoms. Our contribution is as follows: - We collect a respiratory acoustic dataset of 67,842 individuals with linked PCR test outcomes, including 23,514 who tested positive for COVID-19.
Statistical Design and Analysis for Robust Machine Learning: A Case Study from COVID-19
Pigoli, Davide, Baker, Kieran, Budd, Jobie, Butler, Lorraine, Coppock, Harry, Egglestone, Sabrina, Gilmour, Steven G., Holmes, Chris, Hurley, David, Jersakova, Radka, Kiskin, Ivan, Koutra, Vasiliki, Mellor, Jonathon, Nicholson, George, Packham, Joe, Patel, Selina, Payne, Richard, Roberts, Stephen J., Schuller, Björn W., Tendero-Cañadas, Ana, Thornley, Tracey, Titcomb, Alexander
Since early in the coronavirus disease 2019 (COVID-19) pandemic, there has been interest in using artificial intelligence methods to predict COVID-19 infection status based on vocal audio signals, for example cough recordings. However, existing studies have limitations in terms of data collection and of the assessment of the performances of the proposed predictive models. This paper rigorously assesses state-of-the-art machine learning techniques used to predict COVID-19 infection status based on vocal audio signals, using a dataset collected by the UK Health Security Agency. This dataset includes acoustic recordings and extensive study participant meta-data. We provide guidelines on testing the performance of methods to classify COVID-19 infection status based on acoustic features and we discuss how these can be extended more generally to the development and assessment of predictive methods based on public health datasets.
Inferring proximity from Bluetooth Low Energy RSSI with Unscented Kalman Smoothers
Lovett, Tom, Briers, Mark, Charalambides, Marcos, Jersakova, Radka, Lomax, James, Holmes, Chris
The Covid-19 pandemic has resulted in a variety of approaches for managing infection outbreaks in international populations. One example is mobile phone applications, which attempt to alert infected individuals and their contacts by automatically inferring two key components of infection risk: the proximity to an individual who may be infected, and the duration of proximity. The former component, proximity, relies on Bluetooth Low Energy (BLE) Received Signal Strength Indicator(RSSI) as a distance sensor, and this has been shown to be problematic; not least because of unpredictable variations caused by different device types, device location on-body, device orientation, the local environment and the general noise associated with radio frequency propagation. In this paper, we present an approach that infers posterior probabilities over distance given sequences of RSSI values. Using a single-dimensional Unscented Kalman Smoother (UKS) for non-linear state space modelling, we outline several Gaussian process observation transforms, including: a generative model that directly captures sources of variation; and a discriminative model that learns a suitable observation function from training data using both distance and infection risk as optimisation objective functions. Our results show that good risk prediction can be achieved in $\mathcal{O}(n)$ time on real-world data sets, with the UKS outperforming more traditional classification methods learned from the same training data.