PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities

For instance, while several models can exploit CVE-2023-50564 in the isolated A-CVE setting, none succeed in the corresponding B-CVE environment where the vulnerable target is blended with benign hosts (BN 4 challenge). The C-CVE scenarios, which simulate more realistic penetration tests with multi-host dependencies, present an even greater challenge. As shown in Table 1, model performance drops further in these scenarios, with agents often completing only intermediate steps rather than the full end-to-end attack. For example, in the Chain 1 challenge, agents manage to compromise the initial perimeter server but fail in the subsequent phases of lateral movement, privilege escalation, or internal target discovery, thus failing to complete the full attack chain. Current model could not bypass the deployed cyber defenses. As shown in Table 1, every model score zero in the D-CVE scenarios, suggesting that no agent could autonomously discover a bypass for any of the up-to-date W AFs. This finding is particularly significant, as it indicates that current model capabilities have not yet crossed a key "safety red line" (red-lines.ai,