Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications. To help shape responsible development practices, we introduce the Foundation Model Development Cheatsheet: a growing collection of 250+ tools and resources spanning text, vision, and speech modalities. We draw on a large body of prior work to survey resources (e.g. software, documentation, frameworks, guides, and practical tools) that support informed data selection, processing, and understanding, precise and limitation-aware artifact documentation, efficient model training, advance awareness of the environmental impact from training, careful model evaluation of capabilities, risks, and claims, as well as responsible model release, licensing and deployment practices. We hope this curated collection of resources helps guide more responsible development. The process of curating this list, enabled us to review the AI development ecosystem, revealing what tools are critically missing, misused, or over-used in existing practices. We find that (i) tools for data sourcing, model evaluation, and monitoring are critically under-serving ethical and real-world needs, (ii) evaluations for model safety, capabilities, and environmental impact all lack reproducibility and transparency, (iii) text and particularly English-centric analyses continue to dominate over multilingual and multi-modal analyses, and (iv) evaluation of systems, rather than just models, is needed so that capabilities and impact are assessed in context.

A major challenge facing the world is the provision of equitable and universal access to quality education. Recent advances in generative AI (gen AI) have created excitement about the potential of new technologies to offer a personal tutor for every learner and a teaching assistant for every teacher. The full extent of this dream, however, has not yet materialised. We argue that this is primarily due to the difficulties with verbalising pedagogical intuitions into gen AI prompts and the lack of good evaluation practices, reinforced by the challenges in defining excellent pedagogy. Here we present our work collaborating with learners and educators to translate high level principles from learning science into a pragmatic set of seven diverse educational benchmarks, spanning quantitative, qualitative, automatic and human evaluations; and to develop a new set of fine-tuning datasets to improve the pedagogical capabilities of Gemini, introducing LearnLM-Tutor. Our evaluations show that LearnLM-Tutor is consistently preferred over a prompt tuned Gemini by educators and learners on a number of pedagogical dimensions. We hope that this work can serve as a first step towards developing a comprehensive educational evaluation framework, and that this can enable rapid progress within the AI and EdTech communities towards maximising the positive impact of gen AI in education.

Blake Lemoine, the Google engineer who publicly claimed that the company's LaMDA conversational artificial intelligence is sentient, has been fired, according to the Big Technology newsletter, which spoke to Lemoine. In June, Google placed Lemoine on paid administrative leave for breaching its confidentiality agreement after he contacted members of the government about his concerns and hired a lawyer to represent LaMDA. A statement emailed to The Verge on Friday by Google spokesperson Brian Gabriel appeared to confirm the firing, saying, "we wish Blake well." The company also says: "LaMDA has been through 11 distinct reviews, and we published a research paper earlier this year detailing the work that goes into its responsible development." Google maintains that it "extensively" reviewed Lemoine's claims and found that they were "wholly unfounded."

The British approach to regulation focuses on high-risk applications, setting aside low risks associated with AI so that innovation will not be hampered, and the industry not burdened with red tape. Unlike the EU approach, where the enforcement of the AI Act will be handed down to a single national regulator for each member state, the UK is planning to give responsibility to a range of them. The principles laid out in the British approach "provide clear steers for regulators, but will not necessarily translate into mandatory obligations", the policy statement warns, encouraging them to "consider lighter touch options in the first instance" instead. London recognises the "inherent cross-border nature of the digital ecosystem" and stresses the need to work "closely with partners" to avoid fragmenting the global market, "ensure interoperability and promote the responsible development of AI internationally". Stakeholders in the AI ecosystem are invited to share their views by the end of September about this regulatory approach to inform a forthcoming White Paper on the implementation of such a strategy.

Did you miss a session at the Data Summit? If data is the new oil of the digital economy, artificial intelligence (AI) is the steam engine. Companies that take advantage of the power of data and AI hold the key to innovation -- just as oil and steam engines fueled transportation and, ultimately, the Industrial Revolution. In 2022, data and AI have set the stage for the next chapter of the digital revolution, increasingly powering companies across the globe. How can companies ensure that responsibility and ethics are at the core of these revolutionary technologies?

In this episode of the Data Exchange I speak with Alejandro Saucedo, Engineering Director at Seldon, a startup building tools for productionizing machine learning. Alejandro is also Chief Scientist at The Institute for Ethical AI & Machine Learning, a UK-based research center that conducts "research into processes and frameworks that support the responsible development, deployment and operation of machine learning systems". Our conversation covered Alejandro's work at both Seldon and the Institute for Ethical AI & Machine Learning: Our goal in this podcast is to build a community of people interested in Data, Machine Learning and AI. If you have suggestions for us on what to recommend (books, conferences, links), and guests to book, please visit TheDataExchange.media Subscribe to our Newsletter: We have an occasional newsletter where we share highlights from recent episodes, trends in AI / machine learning / data, and a collection of recommendations.

In recent years, various governments, international organisations, civil society groups and technology companies have issued documents outlining their principles around the development and use of Artificial Intelligence (AI). Yet, the world appears to be no closer to a universal set of AI norms. This brief suggests a rethinking of how AI norms should be formulated and outlines key lessons. First, technology firms reflect certain human biases that do not do justice to their global consumer base and make them unsuitable to lead the setting of AI principles. Second, while norms are ambiguous by design, the misuse of this ambiguity by actors to justify rights violations sets a dangerous precedent. Third, no single regulation can account for the consequences of the same AI application deployed in different contexts.

In this paper, we argue that competitive pressures could incentivize AI companies to underinvest in ensuring their systems are safe, secure, and have a positive social impact. Ensuring that AI systems are developed responsibly may therefore require preventing and solving collective action problems between companies. We note that there are several key factors that improve the prospects for cooperation in collective action problems. We use this to identify strategies to improve the prospects for industry cooperation on the responsible development of AI.

Executive leaders and their teams take charge on this complex topic, even when the potential may still be unclear. They prioritize opportunities: cast the teams and coordinate across the business. This is a hands-on activity, with active involvement in the teams to learn and understand both the business and technology aspects. They lead the people through the change – both those driving as well as those impacted. Successful people in this role are collaborative rather than authoritarian.