SandboxEval: Towards Securing Test Environment for Untrusted Code

Abstract--While large language models (LLMs) are powerful assistants in programming tasks, they may also produce mali cious code. T esting LLM-generated code therefore poses significa nt risks to assessment infrastructure tasked with executing u n-trusted code. T o address these risks, this work focuses on evaluating the security and confidentiality properties of t est environments, reducing the risk that LLM-generated code ma y compromise the assessment infrastructure. We introduce Sa nd-boxEval, a test suite featuring manually crafted test cases that simulate real-world safety scenarios for LLM assessment en vi-ronments in the context of untrusted code execution. The sui te evaluates vulnerabilities to sensitive information expos ure, filesys-tem manipulation, external communication, and other poten tially dangerous operations in the course of assessment activity. We demonstrate the utility of SandboxEval by deploying it on an open-source implementation of Dyff, an established AI asse ssment framework used to evaluate the safety of LLMs at scale. We sho w, first, that the test suite accurately describes limitations placed on an LLM operating under instructions to generate malicious c ode. Second, we show that the test results provide valuable insig hts for developers seeking to harden assessment infrastructur e and identify risks associated with LLM execution activities. There is growing interest in using large language models (LLMs) to assist with code generation due to their ability to produce relevant code for various programming tasks [1, 2, 3 ]. However, using code generated by LLMs involves certain risks, as it may contain subtle bugs or security flaws that are not immediately apparent [4, 5, 6, 7]. For instance, a malicious model developer may intentionally train poisone d LLMs to inject malicious code, subtly manipulating completions to benefit themselves [8, 9, 10]. Through prompt injection, a malicious user may manipulate input to produce harmful outputs [11, 12].