Electrification was, without a doubt, the greatest engineering marvel of the 20th century. The electric motor was invented way back in 1821, and the electrical circuit was mathematically analyzed in 1827. But factory electrification, household electrification, and railway electrification all started slowly several decades later. The field of AI was formally founded in 1956. But it's only now--more than six decades later--that AI is expected to revolutionize the way humanity will live and work in the coming decades.

This article elaborates on a niche aspect of the broader cover story on "Rise of Modern NLP and the Need of Interpretability!"At Embibe, we focus on developing interpretable and explainable Deep Learning systems, and we survey the current state of the art techniques to answer some open questions on linguistic wisdom acquired by NLP models. This article is in continuation of the previous article (Discovering the Encoded Linguistic Knowledge in NLP models) to understand what linguistic knowledge is encoded in NLP models. The previous article covers what is probing, how it is different from multi-task learning, and two types of probes -- representation based probes and attention weights based probes. It also shed light on how a probe task (or auxiliary task) is used to assess the linguistic ability of NLP models trained on some other primary task(s). If this in-depth educational content is useful for you, you can subscribe to our AI research mailing list to be alerted when we release new material.

Online exam proctoring company ProctorU announced earlier this week that it will no longer send artificial intelligence-generated reports of potential student misconduct to institutions without ProctorU staff members first reviewing the footage -- a development raising more questions than it answers about higher education's use of the technology. ProctorU, like many companies offering remote proctoring services, employs AI technology to monitor student movement during tests and flag activity that might indicate a student is cheating. Typical behavior that may be flagged by the AI includes students getting up from their seats, not looking consistently at their screens or talking out loud. Previously, as part of its cheapest proctoring package, ProctorU sent AI-generated incident reports to instructors without staff members reviewing them first. The company offered this package on the condition that any AI-generated reports would be reviewed by instructors.

A successful machine learning (ML) project is about a lot more than just model development and deployment. Machine learning is about the full lifecycle of data. It consists of a complex set of steps and a variety of skills, required to achieve actionable outcomes and deliver business value. The level of complexity involved in the ML lifecycle is part of the reason why good practices and fully integrated tools are in their infancy, even in the present day. Other reasons include a lack of skills, poor scalability of models, and a lack of automation as data scientists often come from several different backgrounds and do not always follow best coding and DevOps practices. Furthermore, data scientists and engineers usually work in silos which results in poor collaboration across the teams.

One of the most amazing dynamics within the DevOps enterprise community is seeing business leaders co-presenting success stories with their technology leadership counterparts. For example, Ken Kennedy (executive vice president and president for Technology and Product at CSG) and Kimberly Johnson (chief operating officer at Fannie Mae) described the achievements of their technology leadership counterparts and why it was important to them. I expect this trend to continue, especially given how COVID-19 has accelerated the rate of digital disruption. I believe this bodes well for all of technology. With the rise of hybrid (remote/in-office) product teams, upskilling and online training initiatives will expand.

Databricks today unveiled a new cloud-based machine learning offering that's designed to give engineer everything they need to build, train, deploy, and manage ML models. The new offering is designed to bridge the gap in existing machine learning products that arises by focusing too much on data engineering, ML model creation, or the deployment aspects of the machine learning cycle, Databricks says. "Many ML platforms fall short because they ignore a key challenge in machine learning: they assume that data are available at high quality and ready for training," Databricks says in its announcement. "That requires data teams to stitch together solutions that are good at data but not AI, with others that are good at AI but not data." To address this gap, Databricks lets users switch between user "experiences" that it exposes, including data science/engineering, SQL analytics, and machine learning experiences, to access tools and features relevant to their everyday workflow.

In a real-world setting biological agents do not have infinite resources to learn new things. It is thus useful to recycle previously acquired knowledge in a way that allows for faster, less resource-intensive acquisition of multiple new skills. Neural networks in the brain are likely not entirely re-trained with new tasks, but how they leverage existing computations to learn new tasks is not well understood. In this work, we study this question in artificial neural networks trained on commonly used neuroscience paradigms. Building on recent work from the multi-task learning literature, we propose two ingredients: (1) network modularity, and (2) learning task primitives. Together, these ingredients form inductive biases we call structural and functional, respectively. Using a corpus of nine different tasks, we show that a modular network endowed with task primitives allows for learning multiple tasks well while keeping parameter counts, and updates, low. We also show that the skills acquired with our approach are more robust to a broad range of perturbations compared to those acquired with other multi-task learning strategies. This work offers a new perspective on achieving efficient multi-task learning in the brain, and makes predictions for novel neuroscience experiments in which targeted perturbations are employed to explore solution spaces.

NLP community is currently investing a lot more research and resources into development of deep learning models than training data. While we have made a lot of progress, it is now clear that our models learn all kinds of spurious patterns, social biases, and annotation artifacts. Algorithmic solutions have so far had limited success. An alternative that is being actively discussed is more careful design of datasets so as to deliver specific signals. This position paper maps out the arguments for and against data curation, and argues that fundamentally the point is moot: curation already is and will be happening, and it is changing the world. The question is only how much thought we want to invest into that process.

Recent explorations of large-scale pre-trained language models (PLMs) such as GPT-3 have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger PLMs. However, training a large-scale PLM requires tremendous amounts of computational resources, which is time-consuming and expensive. In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring the availability of many existing well-trained PLMs. To this end, we explore the question that how can previously trained PLMs benefit training larger PLMs in future. Specifically, we introduce a novel pre-training framework named "knowledge inheritance" (KI), which combines both self-learning and teacher-guided learning to efficiently train larger PLMs. Sufficient experimental results demonstrate the feasibility of our KI framework. We also conduct empirical analyses to explore the effects of teacher PLMs' pre-training settings, including model architecture, pre-training data, etc. Finally, we show that KI can well support lifelong learning and knowledge transfer.