Reproduced under a CC BY 4.0 license. Here are the most tweeted papers that were uploaded onto arXiv during August 2021. Results are powered by Arxiv Sanity Preserver. How to avoid machine learning pitfalls: a guide for academic researchers Michael A. Lones Submitted to arXiv on: 5 August 2021 Abstract: This document gives a concise outline of some of the common mistakes that occur when using machine learning techniques, and what can be done to avoid them. It is intended primarily as a guide for research students, and focuses on issues that are of particular concern within academic research, such as the need to do rigorous comparisons and reach valid conclusions.

DALL-E 2, Stable Diffusion, and others transformed the image generation space. We saw more powerful language models, PaLM, and of course ChatGPT. We saw foundation models being developed for speech, music, proteins, and many other data modalities. And, for the first time, these models are now being widely deployed and utilized by consumers to accomplish a wide breadth of useful tasks. What is clear is that while foundation models have opened up unprecedented new possibilities, they are also still raw, imperfect research artifacts that we do not entirely understand. In 2021, we founded the Center for Research on Foundation Models (CRFM), recognizing the critical role of foundation models. CRFM's mission is to understand and improve foundation models from both a technical and societal perspective.

DALL-E 2, Stable Diffusion, and others transformed the image generation space. We saw more powerful language models, PaLM, and of course ChatGPT. We saw foundation models being developed for speech, music, proteins, and many other data modalities. And, for the first time, these models are now being widely deployed and utilized by consumers to accomplish a wide breadth of useful tasks. What is clear is that while foundation models have opened up unprecedented new possibilities, they are also still raw, imperfect research artifacts that we do not entirely understand. In 2021, we founded the Center for Research on Foundation Models (CRFM), recognizing the critical role of foundation models.

In this post, I will cover the papers and research areas that I found most inspiring. I tried to cover the papers that I was aware of but likely missed many relevant ones. Feel free to highlight them as well as ones that you found inspiring in the comments. Pre-trained models were applied in many different domains and started to be considered critical for ML research [1]. In computer vision, supervised pre-trained models such as Vision Transformer [2] have been scaled up [3] and self-supervised pre-trained models have started to match their performance [4]. The latter have been scaled beyond the controlled environment of ImageNet to random collections of images [5]. In speech, new models have been built based on wav2vec 2.0 [6] such as W2v-BERT [7] as well as more powerful multilingual models such as XLS-R [8]. At the same time, we saw new unified pre-trained models for previously under-researched modality pairs such as for videos and language [9] as well as speech and language [10]. In vision and language, controlled studies shed new light on important components of such multi-modal models [11][12].

If the authors of a recent Stanford report (Bommasani et al., 2021) on the opportunities and risks of "foundation models" are to be believed, these models represent a paradigm shift for AI and for the domains in which they will supposedly be used, including education. Although the name is new (and contested (Field, 2021)), the term describes existing types of algorithmic models that are "trained on broad data at scale" and "fine-tuned" (i.e., adapted) for particular downstream tasks, and is intended to encompass large language models such as BERT or GPT-3 and computer vision models such as CLIP. Such technologies have the potential for harm broadly speaking (e.g., Bender et al., 2021), but their use in the educational domain is particularly fraught, despite the potential benefits for learners claimed by the authors. In section 3.3 of the Stanford report, Malik et al. argue that achieving the goal of providing education for all learners requires more efficient computational approaches that can rapidly scale across educational domains and across educational contexts, for which they argue foundation models are uniquely well-suited. However, evidence suggests that not only are foundation models not likely to achieve the stated benefits for learners, but their use may also introduce new risks for harm.