Evaluating Classification Systems Against Soft Labels with Fuzzy Precision and Recall

The challenge task is about training a sound event detection system using the soft labels, to investigate if leveraging information Classification systems are normally trained by minimizing the from the soft labels is beneficial for the acoustic models. However, cross-entropy between system outputs and reference labels, which the evaluation is done using hard labels and hard metrics. Converting makes the Kullback-Leibler divergence a natural choice for measuring soft labels into binary requires choosing a threshold value, and how closely the system can follow the data. Non-binary references finding a good one is not a trivial task. The most straightforward can arise from various sources, and it is often beneficial to use way is to use 0.5 as the threshold, and this is also how the reference the soft labels for training instead of the binarized data. In addition data for the challenge is binarized. However, as a consequence, six to the cross-entropy based measures, precision and recall provide event classes out of 17 are left out from the evaluation, as there are another perspective for measuring the performance of a classification not enough segments with a soft label value above the threshold.