METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities

Abstract--Large-Language Models (LLMs) have shifted the paradigm of natural language data processing. However, their black-boxed and probabilistic characteristics can lead to potential risks in the quality of outputs in diverse LLM applications. Recent studies have tested Quality Attributes (QAs), such as robustness or fairness, of LLMs by generating adversarial input texts. However, existing studies have limited their coverage of QAs and tasks in LLMs and are difficult to extend. Additionally, these studies have only used one evaluation metric, Attack Success Rate (ASR), to assess the effectiveness of their approaches. We (a) Character-swap perturbation example on sentiment analysis task propose a MEtamorphic Testing for Analyzing LLMs (METAL) framework to address these issues by applying Metamorphic Testing (MT) techniques. This approach facilitates the systematic testing of LLM qualities by defining Metamorphic Relations (MRs), which serve as modularized evaluation metrics. The METAL framework can automatically generate hundreds of MRs from templates that cover various QAs and tasks. In addition, we introduced novel metrics that integrate the ASR method into the semantic qualities of text to assess the effectiveness of MRs accurately. Although there are three perturbations of character swapping, the example shows that the LLM "robustly" generates the The advent of Large-Language Models (LLMs) has transformed equivalent outputs. However, in Figure 1(b), replacing synonyms the landscape of natural language-based data retrieval in the text affects the robustness of toxicity detection results.