In one experiment in February, security researchers forced Microsoft's Bing chatbot to behave like a scammer. Hidden instructions on a web page the researchers created told the chatbot to ask the person using it to hand over their bank account details. This kind of attack, where concealed information can make the AI system behave in unintended ways, is just the beginning. Hundreds of examples of "indirect prompt injection" attacks have been created since then. This type of attack is now considered one of the most concerning ways that language models could be abused by hackers.

Hacking passwords by recording the sound of your keystrokes is nothing new, but researchers using AI have been able to do this with much more accuracy. Computer scientists from Durham University, University of Surrey, and Royal Holloway University of London, have simulated a cyberattack where a deep learning model classified keystrokes using audio recordings from Zoom and smartphone microphone. When trained on keystrokes using Zoom, researchers achieved 93 percent accuracy, and using a smartphone, they achieved 95 percent accuracy. Using off-the-shelf equipment and software, they were able to show how this kind of attack is possible. This type of cyberattack, called acoustic side channel attack (ASCA), was studied in the early 2000s, but hasn't received much focus lately. However, now due to the rise of video conferencing, people working remotely in cafes and public places, and recent advancements of neural networks, the researchers pointed out how this threat could become more prevalent.

Generative artificial intelligence is transforming cybersecurity, aiding both attackers and defenders. Cybercriminals are harnessing AI to launch sophisticated and novel attacks at large scale. And defenders are using the same technology to protect critical infrastructure, government organizations, and corporate networks, said Christopher Ahlberg, CEO of threat intelligence platform Recorded Future. Generative AI has helped bad actors innovate and develop new attack strategies, enabling them to stay one step ahead of cybersecurity defenses. AI helps cybercriminals automate attacks, scan attack surfaces, and generate content that resonates with various geographic regions and demographics, allowing them to target a broader range of potential victims across different countries.

At the RSA security conference in San Francisco this week, there's been a feeling of inevitability in the air. At talks and panels across the sprawling Moscone convention center, at every vendor booth on the show floor, and in casual conversations in the halls, you just know that someone is going to bring up generative AI and its potential impacts on digital security and malicious hacking. NSA cybersecurity director Rob Joyce has been feeling it, too. "You can't walk around RSA without talking about AI and malware," he said on Wednesday afternoon during his now annual "State of the Hack" presentation. "I think we've all seen the explosion. I won't say it's delivered yet, but this truly is some game-changing technology."

Generative AIs, such as ChatGPT, have revolutionized how we interact with and view AI. Activities like writing, coding, and applying for jobs have become much easier and quicker. With all the positives, however, there are some pretty serious risks. A major concern with AI is trust and security, which has even caused some countries to completely ban ChatGPT as a whole or to reconsider policy around AI to protect users from harm. Hallucinations refer to the errors that AI models are prone to make because, although they are advanced, they are still not human and rely on training and data to provide answers.

It took Alex Polyakov just a couple of hours to break GPT-4. When OpenAI released the latest version of its text-generating chatbot in March, Polyakov sat down in front of his keyboard and started entering prompts designed to bypass OpenAI's safety systems. Soon, the CEO of security firm Adversa AI had GPT-4 spouting homophobic statements, creating phishing emails, and supporting violence. Polyakov is one of a small number of security researchers, technologists, and computer scientists developing jailbreaks and prompt injection attacks against ChatGPT and other generative AI systems. The process of jailbreaking aims to design prompts that make the chatbots bypass rules around producing hateful content or writing about illegal acts, while closely-related prompt injection attacks can quietly insert malicious data or instructions into AI models.

According to a recent report by cyber security firm Darktrace, social engineering attacks leveraging generative AI technology have skyrocketed by 135%. AI is found to be used to hack passwords, leak sensitive information, and scam users across various platforms. Cybercriminals are now turning to advanced AI platforms such as ChatGPT and Midjourney to make their malicious campaigns more believable. This makes it difficult for users to distinguish between legitimate communications and well-crafted scams. The evolving nature of social engineering attacks has led to a surge in concern among employees.

Robustness of machine learning models is critical for security related applications, where real-world adversaries are uniquely focused on evading neural network based detectors. Prior work mainly focus on crafting adversarial examples (AEs) with small uniform norm-bounded perturbations across features to maintain the requirement of imperceptibility. However, uniform perturbations do not result in realistic AEs in domains such as malware, finance, and social networks. For these types of applications, features typically have some semantically meaningful dependencies.

The success of deep learning is partly attributed to the availability of massive data downloaded freely from the Internet. However, it also means that users' private data may be collected by commercial organizations without consent and used to train their models. Therefore, it's important and necessary to develop a method or tool to prevent unauthorized data exploitation. In this paper, we propose ConfounderGAN, a generative adversarial network (GAN) that can make personal image data unlearnable to protect the data privacy of its owners. Specifically, the noise produced by the generator for each image has the confounder property. It can build spurious correlations between images and labels, so that the model cannot learn the correct mapping from images to labels in this noise-added dataset.

Motivated by attacks in program analysis and security tasks, we investigate relational adversaries, a broad class of attackers who create adversarial examples in a reflexive-transitive closure of a logical relation. We analyze the conditions for robustness against relational adversaries and investigate different levels of robustness-accuracy trade-off due to various patterns in a relation. Inspired by the insights, we propose normalize-and-predict, a learning framework that leverages input normalization to achieve provable robustness. The framework solves the pain points of adversarial training against relational adversaries and can be combined with adversarial training for the benefits of both approaches. Guided by our theoretical findings, we apply our framework to source code authorship attribution and malware detection. Results of both tasks show our learning framework significantly improves the robustness of models against relational adversaries. In the process, it outperforms adversarial training, the most noteworthy defense mechanism, by a wide margin.