Education
Towards Interactive Language Modeling
ter Hoeve, Maartje, Kharitonov, Evgeny, Hupkes, Dieuwke, Dupoux, Emmanuel
Interaction between caregivers and children plays a critical role in human language acquisition and development. Given this observation, it is remarkable that explicit interaction plays little to no role in artificial language modeling -- which also targets the acquisition of human language, yet by artificial models. Moreover, an interactive approach to language modeling has the potential to make language models substantially more versatile and to considerably impact downstream applications. Motivated by these considerations, we pioneer the space of interactive language modeling. As a first contribution we present a road map in which we detail the steps that need to be taken towards interactive language modeling. We then lead by example and take the first steps on this road map, showing the initial feasibility of our approach. As such, this work aims to be the start of a larger research agenda on interactive language modeling.
Artificial Intelligence Ethics and Safety: practical tools for creating "good" models
The AI Robotics Ethics Society (AIRES) is a non-profit organization founded in 2018 by Aaron Hui to promote awareness and the importance of ethical implementation and regulation of AI. AIRES is now an organization with chapters at universities such as UCLA (Los Angeles), USC (University of Southern California), Caltech (California Institute of Technology), Stanford University, Cornell University, Brown University, and the Pontifical Catholic University of Rio Grande do Sul (Brazil). AIRES at PUCRS is the first international chapter of AIRES, and as such, we are committed to promoting and enhancing the AIRES Mission. Our mission is to focus on educating the AI leaders of tomorrow in ethical principles to ensure that AI is created ethically and responsibly. As there are still few proposals for how we should implement ethical principles and normative guidelines in the practice of AI system development, the goal of this work is to try to bridge this gap between discourse and praxis. Between abstract principles and technical implementation. In this work, we seek to introduce the reader to the topic of AI Ethics and Safety. At the same time, we present several tools to help developers of intelligent systems develop "good" models. This work is a developing guide published in English and Portuguese. Contributions and suggestions are welcome.
Model Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Models
Li, Lei, Lin, Yankai, Ren, Xuancheng, Zhao, Guangxiang, Li, Peng, Zhou, Jie, Sun, Xu
As many fine-tuned pre-trained language models~(PLMs) with promising performance are generously released, investigating better ways to reuse these models is vital as it can greatly reduce the retraining computational cost and the potential environmental side-effects. In this paper, we explore a novel model reuse paradigm, Knowledge Amalgamation~(KA) for PLMs. Without human annotations available, KA aims to merge the knowledge from different teacher-PLMs, each of which specializes in a different classification problem, into a versatile student model. The achieve this, we design a Model Uncertainty--aware Knowledge Amalgamation~(MUKA) framework, which identifies the potential adequate teacher using Monte-Carlo Dropout for approximating the golden supervision to guide the student. Experimental results demonstrate that MUKA achieves substantial improvements over baselines on benchmark datasets. Further analysis shows that MUKA can generalize well under several complicate settings with multiple teacher models, heterogeneous teachers, and even cross-dataset teachers.
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
Xu, Yichong, Zhu, Chenguang, Wang, Shuohang, Sun, Siqi, Cheng, Hao, Liu, Xiaodong, Gao, Jianfeng, He, Pengcheng, Zeng, Michael, Huang, Xuedong
Most of today's AI systems focus on using self-attention mechanisms and transformer architectures on large amounts of diverse data to achieve impressive performance gains. In this paper, we propose to augment the transformer architecture with an external attention mechanism to bring external knowledge and context to bear. By integrating external information into the prediction process, we hope to reduce the need for ever-larger models and increase the democratization of AI systems. We find that the proposed external attention mechanism can significantly improve the performance of existing AI systems, allowing practitioners to easily customize foundation AI models to many diverse downstream applications. In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities. The proposed system, Knowledgeable External Attention for commonsense Reasoning (KEAR), reaches human parity on the open CommonsenseQA research benchmark with an accuracy of 89.4\% in comparison to the human accuracy of 88.9\%.
Radar trends to watch: November 2021
The design studio Artefact has created a game to teach middle school students about algorithmic bias. Researchers are building large natural language models, potentially the size of GPT-3, to decode the "speech" of whales. A group at Berkeley has built a robot that uses reinforcement learning to teach itself to walk from scratchโi.e., through trial and error. They used two levels of simulation before loading the model into a physical robot. AI is reinventing computers: AI is driving new kinds of CPUs, new "out of the box" form factors (doorbells, appliances), decision-making rather than traditional computation.
DeepMind's AI can untangle knots. But does it guide human intuition?
This article is part of our reviews of AI research papers, a series of posts that explore the latest findings in artificial intelligence. Deep learning can help discover mathematical relations that evade human scientists, a recent paper by researchers at DeepMind shows. Like many things coming from the Alphabet-owned artificial intelligence lab, the paper, which is titled "Advancing mathematics by guiding human intuition with AI," has received much attention from science and tech media. Some mathematicians and computer scientists have lauded DeepMind's efforts and the findings in the paper as breakthroughs. Others are more skeptical and believe that the use of deep learning in mathematics might have been overstated in the paper and its coverage in popular press.