Benchmarking Generative AI Models for Deep Learning Test Input Generation

Abstract--Test Input Generators (TIGs) are crucial to assess the ability of Deep Learning (DL) image classifiers to provide correct predictions for inputs beyond their training and test sets. Recent advancements in Generative AI (GenAI) models have made them a powerful tool for creating and manipulating synthetic images, although these advancements also imply increased complexity and resource demands for training. Models achieve superior performance by generating a higher number of valid, misclassification-inducing inputs. Figure 1 shows three misclassified inputs for an handwritten digit classifier: input (a) is valid and the expected label (i.e., 9) matches with I. This advancement has enabled raw inputs (i.e., pixel perturbations [16]-[19]) or greater automation, especially in life-and safety-critical areas parametrized semantic representations (e.g., control points in such as healthcare and autonomous driving [2]-[5]. However, these approaches are restricted to modifying as it is difficult to assess their ability to generalize to unseen initial images with known ground-truth labels, limiting exploration data. In fact, their training and test sets may not fully capture to regions near the original inputs and leaving significant the range of real-world scenarios they will encounter after portions of the input space untested. A significant challenge for software Researchers have recently started leveraging the creativity of testers is generating test images that accurately reflect realworld distribution-aware Generative AI (GenAI) models [15], [24]- operating conditions and trigger misclassifications, i.e., [27], which learn the input data distribution in the form of a unexpected behaviors where predicted labels deviate from the latent space, i.e., a compressed low-dimensional representation expected ones.