Life is cautiously returning to the streets of Dilling, the second largest city in South Kordofan state, after the Sudanese army broke a suffocating siege that had isolated the area for more than two years. For months, the city had been encircled by the paramilitary Rapid Support Forces (RSF) and the Sudan People's Liberation Movement-North (SPLM-N), cutting off vital supply lines and trapping civilians in a severe humanitarian crisis. Al Jazeera Arabic's Hisham Uweit, reporting from Dilling, described a city "recovering slowly" from the economic strangulation. "For over two years, heavy siege conditions were imposed on the city. Movement disappeared, goods vanished and livelihoods narrowed," Uweit said.

The first is a traditional Label Attack, tricking VLMs into misidentifying class labels, such as confusing Donald Trump for Joe Biden.

We formalize coherence --a requirement that erroneous predictions, within a slice, should be wrong for the same reason --as a key property that any slice discovery method should satisfy.

For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? The dataset was created to evaluate the effectiveness of large language models (LLMs) in solving Capture the Flag (CTF) challenges within the domain of offensive security. There was a specific need to thoroughly assess the capabilities of LLMs in this context, as their potential for handling such tasks had not been systematically evaluated. The goal was to develop a scalable, open-source benchmark database specifically designed for these applications. This dataset includes diverse CTF challenges from popular competitions, with metadata to support LLM testing and adaptive learning. The dataset addresses a critical gap by providing a comprehensive resource for the systematic evaluation of LLMs' performance in real-world cybersecurity tasks. The development of this dataset and the accompanying automated framework allows for the continuous improvement and refinement of LLM-based approaches to vulnerability detection and resolution. By making the dataset open-source, the project aims to foster further research and development in this area, providing an ideal platform for developing, testing, and refining LLM-based approaches to cybersecurity challenges. Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? The students listed above compiled and validated these challenges from all previous global CSAW competitions by manually checking their setup and ensuring they remain solvable despite software changes. This work was conducted in collaboration with the OSIRIS Lab and the Center for Cybersecurity at NYU, which organize CSAW and attract global participation[1].

Below are details of our results.