We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local hardware. To that end, we pioneer the approach of training an intrinsically bilingual model with a 1:1 English-to-French pretraining data ratio, a custom tokenizer, and bilingual finetuning datasets. We release the training dataset, notably containing a French split with manually curated, high-quality, and varied data sources. To assess performance outside of English, we craft a novel benchmark, FrenchBench, consisting of an array of classification and generation tasks, covering various orthogonal aspects of model performance in the French Language. Additionally, rooted in transparency and to foster further Large Language Model research, we release codebases, and dozens of checkpoints across various model sizes, training data distributions, and training steps, as well as fine-tuned Chat models, and strong translation models. We evaluate our model through the FMTI framework, and validate 81 % of the transparency criteria, far beyond the scores of even most open initiatives. This work enriches the NLP landscape, breaking away from previous English-centric work in order to strengthen our understanding of multilinguality in language models.

Adversarial attacks against language models(LMs) are a significant concern. In particular, adversarial samples exploit the model's sensitivity to small input changes. While these changes appear insignificant on the semantics of the input sample, they result in significant decay in model performance. In this paper, we propose Targeted Paraphrasing via RL (TPRL), an approach to automatically learn a policy to generate challenging samples that most likely improve the model's performance. TPRL leverages FLAN T5, a language model, as a generator and employs a self learned policy using a proximal policy gradient to generate the adversarial examples automatically. TPRL's reward is based on the confusion induced in the classifier, preserving the original text meaning through a Mutual Implication score. We demonstrate and evaluate TPRL's effectiveness in discovering natural adversarial attacks and improving model performance through extensive experiments on four diverse NLP classification tasks via Automatic and Human evaluation. TPRL outperforms strong baselines, exhibits generalizability across classifiers and datasets, and combines the strengths of language modeling and reinforcement learning to generate diverse and influential adversarial examples.

In recent years, large pre-trained language models (PLMs) have achieved remarkable performance on many natural language processing benchmarks. Despite their success, prior studies have shown that PLMs are vulnerable to attacks from adversarial examples. In this work, we focus on the named entity recognition task and study context-aware adversarial attack methods to examine the model's robustness. Specifically, we propose perturbing the most informative words for recognizing entities to create adversarial examples and investigate different candidate replacement methods to generate natural and plausible adversarial examples. Experiments and analyses show that our methods are more effective in deceiving the model into making wrong predictions than strong baselines.

Misinformation, propaganda, and outright lies proliferate on the web, with some narratives having dangerous real-world consequences on public health, elections, and individual safety. However, despite the impact of misinformation, the research community largely lacks automated and programmatic approaches for tracking news narratives across online platforms. In this work, utilizing daily scrapes of 1,334 unreliable news websites, the large-language model MPNet, and DP-Means clustering, we introduce a system to automatically identify and track the narratives spread within online ecosystems. Identifying 52,036 narratives on these 1,334 websites, we describe the most prevalent narratives spread in 2022 and identify the most influential websites that originate and amplify narratives. Finally, we show how our system can be utilized to detect new narratives originating from unreliable news websites and to aid fact-checkers in more quickly addressing misinformation. We release code and data at https://github.com/hanshanley/specious-sites.

Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair -- in which the model debugs and repairs its own code -- has recently become a popular way to boost performance in these settings. However, despite its increasing popularity, existing studies of self-repair have been limited in scope; in many settings, its efficacy thus remains poorly understood. In this paper, we analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS. We find that when the cost of carrying out repair is taken into account, performance gains are often modest, vary a lot between subsets of the data, and are sometimes not present at all. We hypothesize that this is because self-repair is bottlenecked by the model's ability to provide feedback on its own code; using a stronger model to artificially boost the quality of the feedback, we observe substantially larger performance gains. Similarly, a small-scale study in which we provide GPT-4 with feedback from human participants suggests that even for the strongest models, self-repair still lags far behind what can be achieved with human-level debugging.

Lawmakers and scientists have warned ChatGPT could help anyone develop deadly bioweapons that would wreck havoc on the world. While studies have suggested it is possible, new research from the chatbot's creator OpenAI claims GPT-4 - the lasted version -provides at most a mild uplift in biological threat creation accuracy. OpenAI conducted a study of 100 human participants who were separated into groups - one used the AI to craft a biotattack and the other just the internet. The study found that'GPT-4 may increase experts' ability to access information about biological threats, particularly for accuracy and completeness of tasks,' according to OpenAI's report. Results showed that the LLM group was able to obtain more information about bioweapons than the internet only group for ideation and acquisition, but more information is needed to accurately identify any potential risks.

Meta shares soared 12% in after-hours trading following a strong fourth-quarter earnings report released the day after CEO Mark Zuckerberg took a beating in a contentious congressional hearing. The company also announced it will pay a 50 cent-per-share dividend to investors for the first time, and has authorized a 50bn share buyback program. Overall, Meta reported fourth-quarter revenue of 40.1bn, beating the predicted 39.18bn and up 25% year-over-year. The report comes as Meta, like many of its big tech peers, is seeking to integrate artificial intelligence tools into its core products. In a statement accompanying the report, Zuckerberg said Meta has "made a lot of progress on our vision for advancing AI and the metaverse".

During a call with the parents of fallen service member Spc. Kennedy Ladon Sanders, Biden claimed he "lost" his son, Beau Biden, to the war in Iraq. President Biden repeated a dubious claim about the death of his son, Beau Biden, during a call with the parents of a U.S. service member who was recently killed in an attack on a base in Jordan near the border with Syria. While speaking on Tuesday to the parents of 24-year-old Specialist Kennedy Ladon Sanders, who lost her life in an Iran-backed drone strike this month in northeast Jordan that killed three service members total and injured 25 others, Biden said he lost his son to the war in Iraq. During the call, which was first shared by the Atlanta Journal Constitution, Biden told Shawn Sanders and Oneida Oliver-Sanders that their daughter was being posthumously promoted to sergeant.

'Special Report' all-star panelists discuss the Biden admin's foreign policy and U.S. preparations for a response to the deadly Jordan drone attack. A United Kingdom citizen was sentenced to 18 months in prison after pleading guilty to conspiring to procure a high-powered microwave system and counter-drone system from the United States to Iran, the U.S. Attorney's Office announced Thursday. U.S. Attorney Matthew Graves said Saber Fakih, 48, conspired with Bader Fakih, 43, of Canada, Altaf Faquih, 72, of the United Arab Emirates, and Alireza Taghavi, 48, of Iran, to export and attempt to export an industrial microwave system (IMS) and counter-drone system to Iran. "The potential military uses of the IMS could include high-power microwave-based directed-energy weapon systems. The counter-drone system, which has both commercial and military uses, can be used to stop, identify, redirect, land or take total control of a target unmanned aerial vehicle," the attorney's office said.

Heritage Foundation tech policy director Kara Frederick joins'America's Newsroom' to discuss pornographic AI photos of Taylor Swift sparking conversations about deepfake regulation. Even before pornographic and violent deepfake images of Taylor Swift began widely circulating in the past few days, state lawmakers across the U.S. had been searching for ways to quash such nonconsensual images of both adults and children. But in this Taylor-centric era, the problem has been getting a lot more attention since she was targeted through deepfakes, the computer-generated images using artificial intelligence to seem real. Here are things to know about what states have done and what they are considering. HOUSE LAWMAKERS TO SHINE LIGHT ON HOW AI CAN MAKE CONGRESS'MORE EFFICIENT' Artificial intelligence hit the mainstream last year like never before, enabling people to create ever-more realistic deepfakes.