Studying how people interact with large language models (LLMs) in real-world scenarios is increasingly important due to their widespread use in various applications. In this paper, we introduce LMSYS-Chat-1M, a large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs. This dataset is collected from 210K unique IP addresses in the wild on our Vicuna demo and Chatbot Arena website. We offer an overview of the dataset's content, including its curation process, basic statistics, and topic distribution, highlighting its diversity, originality, and scale. We demonstrate its versatility through four use cases: developing content moderation models that perform similarly to GPT-4, building a safety benchmark, training instruction-following models that perform similarly to Vicuna, and creating challenging benchmark questions. We believe that this dataset will serve as a valuable resource for understanding and advancing LLM capabilities. The dataset is publicly available at https://huggingface.co/datasets/lmsys/lmsys-chat-1m.

We introduce Jais and Jais-chat, new state-of-the-art Arabic-centric foundation and instruction-tuned open generative large language models (LLMs). The models are based on the GPT-3 decoder-only architecture and are pretrained on a mixture of Arabic and English texts, including source code in various programming languages. With 13 billion parameters, they demonstrate better knowledge and reasoning capabilities in Arabic than any existing open Arabic and multilingual models by a sizable margin, based on extensive evaluation. Moreover, the models are competitive in English compared to English-centric open models of similar size, despite being trained on much less English data. We provide a detailed description of the training, the tuning, the safety alignment, and the evaluation of the models. We release two open versions of the model -- the foundation Jais model, and an instruction-tuned Jais-chat variant -- with the aim of promoting research on Arabic LLMs. Available at https://huggingface.co/inception-mbzuai/jais-13b-chat

Reinforcement learning (RL) for physical design of silicon chips in a Google 2021 Nature paper stirred controversy due to poorly documented claims that raised eyebrows and drew critical media coverage. The paper withheld critical methodology steps and most inputs needed to reproduce results. Our meta-analysis shows how two separate evaluations filled in the gaps and demonstrated that Google RL lags behind (i) human designers, (ii) a well-known algorithm (Simulated Annealing), and (iii) generally-available commercial software, while being slower; and in a 2023 open research contest, RL methods weren't in top 5. Crosschecked data indicate that the integrity of the Nature paper is substantially undermined owing to errors in conduct, analysis and reporting. Before publishing, Google rebuffed internal allegations of fraud. We note policy implications and conclusions for chip design.

The United States Justice Department (DOJ) is being asked to investigate whether a gunshot-detection system widely in use across the US is being selectively deployed to justify the over-policing of mainly Black neighborhoods, as critics of the technology claim. Attorneys for the nonprofit Electronic Privacy Information Center--a leading US-based civil liberties group--argue that "substantial evidence" suggests American cities are disproportionately deploying an acoustic tool known as ShotSpotter in majority-minority neighborhoods. Citing past studies, EPIC alleges that data derived from these sensors has encouraged some police departments to spend more and more time patrolling areas where the fewest number of white residents live--an allegation disputed by SoundThinking, the system's manufacturer. In a letter today to Merrick Garland, the US attorney general, attorneys for EPIC call for an investigation into whether cities using ShotSpotter are running afoul of the Civil Rights Act--namely, Title VI, which forbids racial discrimination by anyone who receives federal funds. "State and local police departments around the country have used federal financial assistance to facilitate the purchase of a slew of surveillance and automated decision-making technologies, including ShotSpotter," EPIC says.

ChatGPT can now browse the internet to provide users with current information, its parent company OpenAI has announced. The chatbot was previously trained to use data up to September 2021 and was unable to provide real-time information. On Wednesday, Microsoft-backed OpenAI announced on X, formerly Twitter, that the new update allows it to move past the September 2021 cutoff and access current information on the internet. ChatGPT can now browse the internet to provide you with current and authoritative information, complete with direct links to sources. It is no longer limited to data before September 2021.

Japanese police said Thursday they will introduce artificial intelligence technology to identify social media posts through which people are recruited to commit crimes like robbery and fraud. Starting Friday, the National Police Agency will use AI to look for posts promising large payments for yami baito, an expression implying shadowy illegal work, coupled with wordings that solicit people to conduct other more specific criminal acts such as transporting or receiving money obtained via fraudulent means. The concept of yami baito grabbed headlines in Japan recently after a group of Japanese men, arrested earlier this year for running scams from the Philippines, was alleged to have recruited individuals via social media to carry out a series of robberies across Japan, with at least one resulting in a murder.

As machine learning models are increasingly being employed in various high-stakes settings, it becomes important to ensure that predictions of these models are not only adversarially robust, but also readily explainable to relevant stakeholders. However, it is unclear if these two notions can be simultaneously achieved or if there exist trade-offs between them. In this work, we make one of the first attempts at studying the impact of adversarially robust models on actionable explanations which provide end users with a means for recourse. We theoretically and empirically analyze the cost (ease of implementation) and validity (probability of obtaining a positive model prediction) of recourses output by state-of-the-art algorithms when the underlying models are adversarially robust vs. non-robust. More specifically, we derive theoretical bounds on the differences between the cost and the validity of the recourses generated by state-of-the-art algorithms for adversarially robust vs. non-robust linear and non-linear models. Our empirical results with multiple real-world datasets validate our theoretical results and show the impact of varying degrees of model robustness on the cost and validity of the resulting recourses. Our analyses demonstrate that adversarially robust models significantly increase the cost and reduce the validity of the resulting recourses, thus shedding light on the inherent trade-offs between adversarial robustness and actionable explanations.

Large language models (LLMs) have demonstrated strong capabilities in various aspects. However, when applying them to the highly specialized, safe-critical legal domain, it is unclear how much legal knowledge they possess and whether they can reliably perform legal-related tasks. To address this gap, we propose a comprehensive evaluation benchmark LawBench. LawBench has been meticulously crafted to have precise assessment of the LLMs' legal capabilities from three cognitive levels: (1) Legal knowledge memorization: whether LLMs can memorize needed legal concepts, articles and facts; (2) Legal knowledge understanding: whether LLMs can comprehend entities, events and relationships within legal text; (3) Legal knowledge applying: whether LLMs can properly utilize their legal knowledge and make necessary reasoning steps to solve realistic legal tasks. LawBench contains 20 diverse tasks covering 5 task types: single-label classification (SLC), multi-label classification (MLC), regression, extraction and generation. We perform extensive evaluations of 51 LLMs on LawBench, including 20 multilingual LLMs, 22 Chinese-oriented LLMs and 9 legal specific LLMs. The results show that GPT-4 remains the best-performing LLM in the legal domain, surpassing the others by a significant margin. While fine-tuning LLMs on legal specific text brings certain improvements, we are still a long way from obtaining usable and reliable LLMs in legal tasks. All data, model predictions and evaluation code are released in https://github.com/open-compass/LawBench/. We hope this benchmark provides in-depth understanding of the LLMs' domain-specified capabilities and speed up the development of LLMs in the legal domain.

Large language models (LLMs) have demonstrated human-level performance on a vast spectrum of natural language tasks. However, few studies have addressed the LLM threat and vulnerability from an ideology perspective, especially when they are increasingly being deployed in sensitive domains, e.g., elections and education. In this study, we explore the implications of GPT soft ideologization through the use of AI-self-consciousness. By utilizing GPT self-conversations, AI can be granted a vision to "comprehend" the intended ideology, and subsequently generate finetuning data for LLM ideology injection. When compared to traditional government ideology manipulation techniques, such as information censorship, LLM ideologization proves advantageous; it is easy to implement, cost-effective, and powerful, thus brimming with risks.

Kurt "The Cyberguy" Knutsson explains the benefits of using the new AI massage bot. Imagine stumbling upon a video of yourself doing something you've never done or saying something you've never said. That's the unsettling reality many face with the surge of deepfakes, and celebrities are the prime targets. In an era swarming with unauthorized AI-generated content, one startup is stepping up to help celebs keep control of their own images, voices and performance data. Metaphysic, already recognized for its convincing deepfake videos, has launched a new tool, Metaphysic Pro.