Education
ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses for Solving Undergraduate Computer Science Questions
Joshi, Ishika, Budhiraja, Ritvik, Dev, Harshal, Kadia, Jahnvi, Ataullah, M. Osama, Mitra, Sayan, Kumar, Dhruv, Akolekar, Harshal D.
ChatGPT is an AI language model developed by OpenAI that can understand and generate human-like text. It can be used for a variety of use cases such as language generation, question answering, text summarization, chatbot development, language translation, sentiment analysis, content creation, personalization, text completion, and storytelling. While ChatGPT has garnered significant positive attention, it has also generated a sense of apprehension and uncertainty in academic circles. There is concern that students may leverage ChatGPT to complete take-home assignments and exams and obtain favorable grades without genuinely acquiring knowledge. This paper adopts a quantitative approach to demonstrate ChatGPT's high degree of unreliability in answering a diverse range of questions pertaining to topics in undergraduate computer science. Our analysis shows that students may risk self-sabotage by blindly depending on ChatGPT to complete assignments and exams. We build upon this analysis to provide constructive recommendations to both students and instructors.
Learning Robust, Agile, Natural Legged Locomotion Skills in the Wild
Wang, Yikai, Jiang, Zheyuan, Chen, Jianyu
Recently, reinforcement learning has become a promising and polular solution for robot legged locomotion. Compared to model-based control, reinforcement learning based controllers can achieve better robustness against uncertainties of environments through sim-to-real learning. However, the corresponding learned gaits are in general overly conservative and unatural. In this paper, we propose a new framework for learning robust, agile and natural legged locomotion skills over challenging terrain. We incorporate an adversarial training branch based on real animal locomotion data upon a teacher-student training pipeline for robust sim-to-real transfer. Empirical results on both simulation and real world of a quadruped robot demonstrate that our proposed algorithm enables robustly traversing challenging terrains such as stairs, rocky ground and slippery floor with only proprioceptive perception. Meanwhile, the gaits are more agile, natural, and energy efficient compared to the baselines. Both qualitative and quantitative results are presented in this paper.
Viral 'AI yearbook' trend allows people to create nostalgic high-school photos from any era
Fox News correspondent Grady Trimble has the latest on fears the technology will spiral out of control on'Special Report.' Now everyone can be the coolest kid in school. Using artificial intelligence, people are turning themselves into nostalgic high school personas in a new and already-viral "AI yearbook" trend. Social media users have tried out AI-powered platforms such as the EPIK AI photo editor, which allows people to turn a modern photo of themselves into what looks like a yearbook portrait from any era (the 1990s has been the most popular). TikTok influencers such as Olivia Dunne and Joe Mele have also used these apps to transform themselves into classic high-school tropes like the jock, the nerd, the rebel, the popular kid and other stereotypes.
As Japan releases more Fukushima water, what about the rest of the plant?
Before the 2011 tsunami inundated Ukedo elementary school's classrooms, the ocean was central to the school's identity. In the summer, pupils would run down the 300-metre path to the beach, splitting up into groups to see who could make the best animals out of sand. Every year, students also painted local fishermen's boats, a tradition that resonated strongly in Namie town, where many parents worked in the fishing industry. But when a magnitude 9.0 earthquake, a subsequent tsunami and a nuclear disaster brought devastation to Japan's northeastern Tohoku region, that all changed, Shinichi Sato, a teacher who taught at Ukedo elementary school, told Al Jazeera. "For years after the disaster, we weren't allowed to teach lessons outside, in fear that kids would touch radioactive soil," Sato said.
Artificial Intelligence Index Report 2023
Maslej, Nestor, Fattorini, Loredana, Brynjolfsson, Erik, Etchemendy, John, Ligett, Katrina, Lyons, Terah, Manyika, James, Ngo, Helen, Niebles, Juan Carlos, Parli, Vanessa, Shoham, Yoav, Wald, Russell, Clark, Jack, Perrault, Raymond
Welcome to the sixth edition of the AI Index Report! This year, the report introduces more original data than any previous edition, including a new chapter on AI public opinion, a more thorough technical performance chapter, original analysis about large language and multimodal models, detailed trends in global AI legislation records, a study of the environmental impact of AI systems, and more. The AI Index Report tracks, collates, distills, and visualizes data related to artificial intelligence. Our mission is to provide unbiased, rigorously vetted, broadly sourced data in order for policymakers, researchers, executives, journalists, and the general public to develop a more thorough and nuanced understanding of the complex field of AI. The report aims to be the world's most credible and authoritative source for data and insights about AI.
Design Principles for Lifelong Learning AI Accelerators
Kudithipudi, Dhireesha, Daram, Anurag, Zyarah, Abdullah M., Zohora, Fatima Tuz, Aimone, James B., Yanguas-Gil, Angel, Soures, Nicholas, Neftci, Emre, Mattina, Matthew, Lomonaco, Vincenzo, Thiem, Clare D., Epstein, Benjamin
Lifelong learning - an agent's ability to learn throughout its lifetime - is a hallmark of biological learning systems and a central challenge for artificial intelligence (AI). The development of lifelong learning algorithms could lead to a range of novel AI applications, but this will also require the development of appropriate hardware accelerators, particularly if the models are to be deployed on edge platforms, which have strict size, weight, and power constraints. Here, we explore the design of lifelong learning AI accelerators that are intended for deployment in untethered environments. We identify key desirable capabilities for lifelong learning accelerators and highlight metrics to evaluate such accelerators. We then discuss current edge AI accelerators and explore the future design of lifelong learning accelerators, considering the role that different emerging technologies could play.
BooookScore: A systematic exploration of book-length summarization in the era of LLMs
Chang, Yapei, Lo, Kyle, Goyal, Tanya, Iyyer, Mohit
Summarizing book-length documents (>100K tokens) that exceed the context window size of large language models (LLMs) requires first breaking the input document into smaller chunks and then prompting an LLM to merge, update, and compress chunk-level summaries. Despite the complexity and importance of this task, it has yet to be meaningfully studied due to the challenges of evaluation: existing book-length summarization datasets (e.g., BookSum) are in the pretraining data of most public LLMs, and existing evaluation methods struggle to capture errors made by modern LLM summarizers. In this paper, we present the first study of the coherence of LLM-based book-length summarizers implemented via two prompting workflows: (1) hierarchically merging chunk-level summaries, and (2) incrementally updating a running summary. We obtain 1193 fine-grained human annotations on GPT-4 generated summaries of 100 recently-published books and identify eight common types of coherence errors made by LLMs. Because human evaluation is expensive and time-consuming, we develop an automatic metric, BooookScore, that measures the proportion of sentences in a summary that do not contain any of the identified error types. BooookScore has high agreement with human annotations and allows us to systematically evaluate the impact of many other critical parameters (e.g., chunk size, base LLM) while saving $15K and 500 hours in human evaluation costs. We find that closed-source LLMs such as GPT-4 and Claude 2 produce summaries with higher BooookScore than the oft-repetitive ones generated by LLaMA 2. Incremental updating yields lower BooookScore but higher level of detail than hierarchical merging, a trade-off sometimes preferred by human annotators. We release code and annotations after blind review to spur more principled research on book-length summarization.
Using Large Language Models for Qualitative Analysis can Introduce Serious Bias
Ashwin, Julian, Chhabra, Aditya, Rao, Vijayendra
Large Language Models (LLMs) are quickly becoming ubiquitous, but the implications for social science research are not yet well understood. This paper asks whether LLMs can help us analyse large-N qualitative data from open-ended interviews, with an application to transcripts of interviews with Rohingya refugees in Cox's Bazaar, Bangladesh. We find that a great deal of caution is needed in using LLMs to annotate text as there is a risk of introducing biases that can lead to misleading inferences. We here mean bias in the technical sense, that the errors that LLMs make in annotating interview transcripts are not random with respect to the characteristics of the interview subjects. Training simpler supervised models on high-quality human annotations with flexible coding leads to less measurement error and bias than LLM annotations. Therefore, given that some high quality annotations are necessary in order to asses whether an LLM introduces bias, we argue that it is probably preferable to train a bespoke model on these annotations than it is to use an LLM for annotation.
Towards practical reinforcement learning for tokamak magnetic control
Tracey, Brendan D., Michi, Andrea, Chervonyi, Yuri, Davies, Ian, Paduraru, Cosmin, Lazic, Nevena, Felici, Federico, Ewalds, Timo, Donner, Craig, Galperti, Cristian, Buchli, Jonas, Neunert, Michael, Huber, Andrea, Evens, Jonathan, Kurylowicz, Paula, Mankowitz, Daniel J., Riedmiller, Martin, Team, The TCV
Reinforcement learning (RL) has shown promising results for real-time control systems, including the domain of plasma magnetic control. However, there are still significant drawbacks compared to traditional feedback control approaches for magnetic confinement. In this work, we address key drawbacks of the RL method; achieving higher control accuracy for desired plasma properties, reducing the steady-state error, and decreasing the required time to learn new tasks. We build on top of \cite{degrave2022magnetic}, and present algorithmic improvements to the agent architecture and training procedure. We present simulation results that show up to 65\% improvement in shape accuracy, achieve substantial reduction in the long-term bias of the plasma current, and additionally reduce the training time required to learn new tasks by a factor of 3 or more. We present new experiments using the upgraded RL-based controllers on the TCV tokamak, which validate the simulation results achieved, and point the way towards routinely achieving accurate discharges using the RL approach.
PyTrial: Machine Learning Software and Benchmark for Clinical Trial Applications
Wang, Zifeng, Theodorou, Brandon, Fu, Tianfan, Xiao, Cao, Sun, Jimeng
Clinical trials are conducted to test the effectiveness and safety of potential drugs in humans for regulatory approval. Machine learning (ML) has recently emerged as a new tool to assist in clinical trials. Despite this progress, there have been few efforts to document and benchmark ML4Trial algorithms available to the ML research community. Additionally, the accessibility to clinical trial-related datasets is limited, and there is a lack of well-defined clinical tasks to facilitate the development of new algorithms. To fill this gap, we have developed PyTrial that provides benchmarks and open-source implementations of a series of ML algorithms for clinical trial design and operations. In this paper, we thoroughly investigate 34 ML algorithms for clinical trials across 6 different tasks, including patient outcome prediction, trial site selection, trial outcome prediction, patient-trial matching, trial similarity search, and synthetic data generation. We have also collected and prepared 23 ML-ready datasets as well as their working examples in Jupyter Notebooks for quick implementation and testing. PyTrial defines each task through a simple four-step process: data loading, model specification, model training, and model evaluation, all achievable with just a few lines of code. Furthermore, our modular API architecture empowers practitioners to expand the framework to incorporate new algorithms and tasks effortlessly. The code is available at https://github.com/RyanWangZf/PyTrial.