AI red teaming brings security thinking to LLM applications by probing failure modes such as prompt injection, output manipulation, and sensitive data exfiltration. While automated and curated benchmarks (e.g., JailbreakBench [1], HarmBench [2]) are increasingly used to test models and defenses, comparatively fewer studies analyze community scale behavior in the wild. We study ai_gon3_rogu3 [3], a 10 day competition with 504 registrants and 217 active players, to quantify solve dynamics, tactic stratification, and choke points across 11 challenges. We find sharp skill stratification, higher success for output manipulation than for data extraction, and strong effects of format obfuscation tactics, with multi step defenses remaining robust, among other insights.

The rapid and dynamic pace of Artificial Intelligence (AI) and Machine Learning (ML) is revolutionizing the insurance sector. AI offers significant, very much welcome advantages to insurance companies, and is fundamental to their customer-centricity strategy. It also poses challenges, in the project and implementation phase. Among those, we study Adversarial Attacks, which consist of the creation of modified input data to deceive an AI system and produce false outputs. We provide examples of attacks on insurance AI applications, categorize them, and argue on defence methods and precautionary systems, considering that they can involve few-shot and zero-shot multilabelling. A related topic, with growing interest, is the validation and verification of systems incorporating AI and ML components. These topics are discussed in various sections of this paper.

The Department of Defense's (DoD) Defense Advanced Research Projects Agency (DARPA) announced the launch of its Guaranteeing AI Robustness Against Deception (GARD) program, which is designed to develop new defenses against adversarial attacks on machine learning (ML) models. The program aims to respond to adversarial AI by developing a testbed to characterize different ML defenses and assess their applicability. Researchers on the program have created resources and virtual tools for the community to be able to test and verify the effectiveness of existing and emerging ML defense models. "Other technical communities – like cryptography – have embraced transparency and found that if you are open to letting people take a run at things, the technology will improve," GARD program manager Bruce Draper said in the announcement. "With GARD, we are taking a page from cryptography and are striving to create a community to facilitate the open exchange of ideas, tools, and technologies that can help researchers test and evaluate their ML defenses. Our goal is to raise the bar on existing evaluation efforts, bringing more sophistication and maturation to the field."

Artificial intelligence (AI) is one of the most discussed technology fields today – and for good reason. AI will soon impact nearly every aspect of our lives and we have only just begun scratching the surface of AI's true potential. With AI, we are deepening our knowledge of human genetics and delivering leaps in medicine, deploying self-driving vehicles and robots for an array of industries, and combating fraud and cybercrime, to name just a few of the growing list of applications. However, as with any nascent technology, AI has the potential to cause harm when placed in the wrong hands. We've begun seeing AI used for nefarious purposes, chiefly in the form of AI-facilitated Cyberattacks, and forecast Adversarial AI to be the next challenge to be faced in this area.

In the last post, we discussed an outline of AI powered cyber attacks and their defence strategies. In this post, we will discuss a specific type of attack which is called adversarial attack. Adversarial attacks are not common now because there are not many deep learning systems in production. But soon, we expect that they will increase. Adversarial attacks are easy to describe.

Stopping ransomware has become a priority for many organizations. So, they are turning to artificial intelligence (AI) and machine learning (ML) as their defenses of choice. However, threat actors are also turning to AI and ML to launch their attacks. One specific type of attack, data poisoning, takes advantage of this. Like any other tech, AI is a two-sided coin.

Artificial intelligence, including machine learning, has added a new element to security operations, too, according to Ollmann.

Artificial intelligence (AI) is a technology that presents great opportunities for many organisations and society as a whole. Many institutions are incorporating AI into their business processes to find efficiencies, improve their decision making, and offer better end user experiences. Even though AI attack surfaces are emerging, future security strategies should take account of adversarial AI, with the emphasis on engineering resilient modelling structures and strengthening against attempts to introduce adversarial manipulation. As adversarial AI has emerged over the past five years, Accenture has seen an increasing number of adversarial attacks exploiting machine learning models. Such exploitation could multiply with the magnitude of threats facing organisations.

Researchers at the University of California at Riverside are working to teach computer vision systems what objects typically exist in close proximity to one another so that if one is altered, the system can flag it, potentially thwarting malicious interference with artificial intelligence systems. The yearlong project, supported by a nearly $1 million grant from the Defense Advanced Research Projects Agency, aims to understand how hackers target machine-vision systems with adversarial AI attacks. Led by Amit Roy-Chowdhury, an electrical and computer engineering professor at the school's Marlan and Rosemary Bourns College of Engineering, the project is part of the Machine Vision Disruption program within DARPA's AI Explorations program. Adversarial AI attacks – which attempt to fool machine learning models by supplying deceptive input -- are gaining attention. "Adversarial attacks can destabilize AI technologies, rendering them less safe, predictable, or reliable," Carnegie Mellon University Professor David Danks wrote in IEEE's Spectrum in February.

An NLP bot gives an erroneous summary of an intercepted wire. These are examples of how AI systems can be hacked, which is an area of increased focus for government and industrial leaders alike. As AI technology matures, it's being adopted widely, which is great. That is what is supposed to happen, after all. However, greater reliance on automated decision-making in the real world brings a greater threat that bad actors will employ techniques like adversarial machine learning and data poisoning to hack our AI systems.