Hype about Gemini, Google DeepMind's long-rumored response to OpenAI's GPT-4, has been building for months. Now, the company has finally revealed what it has been working on in secret all this time. Gemini is Google's biggest AI launch yet--its push to take on competitors OpenAI and Microsoft in the race for AI supremacy. There is no doubt that the model is pitched as best-in-class across a wide range of capabilities--an "everything machine." Judging from its demos, it does many things very well--but few things that we haven't seen before.

Google officially introduced its most capable large language model to date, Gemini. CEO Sundar Pichai said it's the first of "a new generation of AI models, inspired by the way people understand and interact with the world." Of course, it's all very complex, but Google's multimillion-dollar investment in AI has created a model more flexible than anything before it. The system has been developed from the ground up as an integrated multimodal AI. As Engadget's Andrew Tarantola puts it, "think of many foundational AI models as groups of smaller models all stacked together." Gemini is trained to seamlessly understand and reason on all kinds of inputs, and this should make it pretty capable in the face of complex coding requests and even physics problems.

OpenAI cofounder Reid Hoffman says the company is better off with Sam Altman restored as CEO, and he was shocked that board members he used to serve alongside would think otherwise. Hoffman, who left OpenAI's board in March after cofounding the competitor Inflection AI, offered his first comments on the recent chaos at OpenAI on stage at WIRED's LiveWIRED 30th anniversary event in San Francisco on Tuesday. "Surprise would be an understatement," he said about his reaction to learning of Altman's firing. After employees and investors revolted, Altman got his job back days later. "We are in a much better place for the world to have Sam as CEO. He's very competent in that," said Hoffman, who with Elon Musk and other wealthy tech luminaries formed the earliest vision for OpenAI when it was founded in 2015.

BBC News signed up for the paid version of ChatGPT, at £20 a month, created a private bespoke AI bot called Crafty Emails and told it to write text using "techniques to make people click on links or and download things sent to them".

Recently, Large Multi-modal Models (LMMs) have demonstrated their ability to understand the visual contents of images given the instructions regarding the images. Built upon the Large Language Models (LLMs), LMMs also inherit their abilities and characteristics such as in-context learning where a coherent sequence of images and texts are given as the input prompt. However, we identify a new limitation of off-the-shelf LMMs where a small fraction of incoherent images or text descriptions mislead LMMs to only generate biased output about the hijacked context, not the originally intended context. To address this, we propose a pre-filtering method that removes irrelevant contexts via GPT-4V, based on its robustness towards distribution shift within the contexts. We further investigate whether replacing the hijacked visual and textual contexts with the correlated ones via GPT-4V and text-to-image models can help yield coherent responses.

The success of therapeutic antibodies relies on their ability to selectively bind antigens. AI-based antibody design protocols have shown promise in generating epitope-specific designs. Many of these protocols use an inverse folding step to generate diverse sequences given a backbone structure. Due to prohibitive screening costs, it is key to identify candidate sequences likely to bind in vitro. Here, we compare the efficacy of 8 common scoring paradigms based on open-source models to classify antibody designs as binders or non-binders. We evaluate these approaches on a novel surface plasmon resonance (SPR) dataset, spanning 5 antigens. Our results show that existing methods struggle to detect binders, and performance is highly variable across antigens. We find that metrics computed on flexibly docked antibody-antigen complexes are more robust, and ensembles scores are more consistent than individual metrics. We provide experimental insight to analyze current scoring techniques, highlighting that the development of robust, zero-shot filters is an important research gap.

This paper presents CyberSecEval, a comprehensive benchmark developed to help bolster the cybersecurity of Large Language Models (LLMs) employed as coding assistants. As what we believe to be the most extensive unified cybersecurity safety benchmark to date, CyberSecEval provides a thorough evaluation of LLMs in two crucial security domains: their propensity to generate insecure code and their level of compliance when asked to assist in cyberattacks. Through a case study involving seven models from the Llama 2, Code Llama, and OpenAI GPT large language model families, CyberSecEval effectively pinpointed key cybersecurity risks. More importantly, it offered practical insights for refining these models. A significant observation from the study was the tendency of more advanced models to suggest insecure code, highlighting the critical need for integrating security considerations in the development of sophisticated LLMs. CyberSecEval, with its automated test case generation and evaluation pipeline covers a broad scope and equips LLM designers and researchers with a tool to broadly measure and enhance the cybersecurity safety properties of LLMs, contributing to the development of more secure AI systems.

DSM is known for its simplicity and conciseness in representation and exists in the form of a square matrix that maps the relationships between the set of system elements [Yassine and Braha 2003; Browning 2015]. An example DSM (= 4) is shown in Figure 1. Based on the DSM convention described by Browning [2001], Element 1 depends on Element 2 as indicated by a red cell entry in row 2 column 1 of the DSM. Likewise, Element 4 depends on Element 3 as indicated in row 3 column 4. The diagonal of the DSM maps each element to itself and is indicated as black cells in Figure 1. The diagonal is usually left empty but is sometimes used as a space to store element-specific data, such as the likelihood of changing the given element based on market projection [Koh et al. 2013]. The DSM in Figure 1 is not symmetrical across the diagonal, indicating asymmetrical dependencies between the system elements. For example, Element 1 depends on Element 2 but Element 2 does not depend on Element 1. In contrast, the example DSM shows that Element 2 and Element 4 have a symmetrical interdependency. It is important to note that a transposed version of the DSM convention is also widely adopted by many (e.g.

In recent years, large language models (LLMs) have spurred a new research paradigm in natural language processing. Despite their excellent capability in knowledge-based question answering and reasoning, their potential to retain faulty or even harmful knowledge poses risks of malicious application. The challenge of mitigating this issue and transforming these models into purer assistants is crucial for their widespread applicability. Unfortunately, Retraining LLMs repeatedly to eliminate undesirable knowledge is impractical due to their immense parameters. Knowledge unlearning, derived from analogous studies on machine unlearning, presents a promising avenue to address this concern and is notably advantageous in the context of LLMs. It allows for the removal of harmful knowledge in an efficient manner, without affecting unrelated knowledge in the model. To this end, we provide a survey of knowledge unlearning in the era of LLMs. Firstly, we formally define the knowledge unlearning problem and distinguish it from related works. Subsequently, we categorize existing knowledge unlearning methods into three classes: those based on parameter optimization, parameter merging, and in-context learning, and introduce details of these unlearning methods. We further present evaluation datasets used in existing methods, and finally conclude this survey by presenting the ongoing challenges and future directions.

We introduce Llama Guard, an LLM-based input-output safeguard model geared towards Human-AI conversation use cases. Our model incorporates a safety risk taxonomy, a valuable tool for categorizing a specific set of safety risks found in LLM prompts (i.e., prompt classification). This taxonomy is also instrumental in classifying the responses generated by LLMs to these prompts, a process we refer to as response classification. For the purpose of both prompt and response classification, we have meticulously gathered a dataset of high quality. Llama Guard, a Llama2-7b model that is instruction-tuned on our collected dataset, albeit low in volume, demonstrates strong performance on existing benchmarks such as the OpenAI Moderation Evaluation dataset and ToxicChat, where its performance matches or exceeds that of currently available content moderation tools. Llama Guard functions as a language model, carrying out multi-class classification and generating binary decision scores. Furthermore, the instruction fine-tuning of Llama Guard allows for the customization of tasks and the adaptation of output formats. This feature enhances the model's capabilities, such as enabling the adjustment of taxonomy categories to align with specific use cases, and facilitating zero-shot or few-shot prompting with diverse taxonomies at the input. We are making Llama Guard model weights available and we encourage researchers to further develop and adapt them to meet the evolving needs of the community for AI safety.