This roundup highlights some interesting NLP papers from October 2022 around language model capabilities. This article's title and TL;DR have been generated with Cohere. Get started with text generation. NLP is evolving at a rapid pace, and every month we discover new capabilities. Large language models, like those built by Cohere, are being used for use cases that we couldn't have imagined even just a few months ago.

The graph represents a network of 2,315 Twitter users whose tweets in the requested range contained "iot ai", or who were replied to or mentioned in those tweets. The network was obtained from the NodeXL Graph Server on Wednesday, 23 November 2022 at 12:29 UTC. The requested start date was Wednesday, 23 November 2022 at 01:01 UTC and the maximum number of tweets (going backward in time) was 7,500. The tweets in the network were tweeted over the 19-day, 4-hour, 44-minute period from Thursday, 03 November 2022 at 20:15 UTC to Wednesday, 23 November 2022 at 01:00 UTC. Additional tweets that were mentioned in this data set were also collected from prior time periods.

How your business can improve the skills of its talent to take greater advantage of AI. There's no doubt that artificial intelligence (AI) is changing the way business is done today. AI will ultimately transform every business in every industry. However, despite their desire to use data science when making decisions, many organizations can't find enough qualified data scientists to develop and run their data science initiatives. Nonetheless, with online training and readily available tools, any software engineer -- or even a business user with a math background -- can become a data scientist.

Public high school in America was the product of the time of its invention, which was way back in 1821. But in this era of rapid technological change marked by artificial intelligence and robots moving into more aspects of work and social life, maybe the way teaching is done in high school needs a reboot. It is framed around the thought experiment: What would an ideal high school of the year 2040 look like? The tour guides of this imagined school of the future are two authors: Jim Tracy, a senior advisor at the nonprofit Jobs for the Future who in his career has led private K-12 schools and served as a college president; and Greg Toppo, longtime education journalist. They instead focus on how coming technological change will end up shifting the relationship between people and machines, and therefore between students and teachers.

Public high school in America was the product of the time of its invention, which was way back in 1821. But in this era of rapid technological change marked by artificial intelligence and robots moving into more aspects of work and social life, maybe the way teaching is done in high school needs a reboot. It is framed around the thought experiment: What would an ideal high school of the year 2040 look like? The tour guides of this imagined school of the future are two authors: Jim Tracy, a senior advisor at the nonprofit Jobs for the Future who in his career has led private K-12 schools and served as a college president; and Greg Toppo, longtime education journalist. They instead focus on how coming technological change will end up shifting the relationship between people and machines, and therefore between students and teachers.

Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness. In this work, however, we reveal that cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. Specifically, we show that many test instances are either over- or under-represented during retrieval, significantly hurting the retrieval performance. To address this problem, we propose Normalized Contrastive Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the instance-wise biases that properly normalize the sum retrieval probabilities of each instance so that every text and video instance is fairly represented during cross-modal retrieval. Empirical study shows that NCL brings consistent and significant gains in text-video retrieval on different model architectures, with new state-of-the-art multimodal retrieval metrics on the ActivityNet, MSVD, and MSR-VTT datasets without any architecture engineering.

The matrix-based R\'enyi's entropy allows us to directly quantify information measures from given data, without explicit estimation of the underlying probability distribution. This intriguing property makes it widely applied in statistical inference and machine learning tasks. However, this information theoretical quantity is not robust against noise in the data, and is computationally prohibitive in large-scale applications. To address these issues, we propose a novel measure of information, termed low-rank matrix-based R\'enyi's entropy, based on low-rank representations of infinitely divisible kernel matrices. The proposed entropy functional inherits the specialty of of the original definition to directly quantify information from data, but enjoys additional advantages including robustness and effective calculation. Specifically, our low-rank variant is more sensitive to informative perturbations induced by changes in underlying distributions, while being insensitive to uninformative ones caused by noises. Moreover, low-rank R\'enyi's entropy can be efficiently approximated by random projection and Lanczos iteration techniques, reducing the overall complexity from $\mathcal{O}(n^3)$ to $\mathcal{O}(n^2 s)$ or even $\mathcal{O}(ns^2)$, where $n$ is the number of data samples and $s \ll n$. We conduct large-scale experiments to evaluate the effectiveness of this new information measure, demonstrating superior results compared to matrix-based R\'enyi's entropy in terms of both performance and computational efficiency.

Partial Label (PL) learning refers to the task of learning from the partially labeled data, where each training instance is ambiguously equipped with a set of candidate labels but only one is valid. Advances in the recent deep PL learning literature have shown that the deep learning paradigms, e.g., self-training, contrastive learning, or class activate values, can achieve promising performance. Inspired by the impressive success of deep Semi-Supervised (SS) learning, we transform the PL learning problem into the SS learning problem, and propose a novel PL learning method, namely Partial Label learning with Semi-supervised Perspective (PLSP). Specifically, we first form the pseudo-labeled dataset by selecting a small number of reliable pseudo-labeled instances with high-confidence prediction scores and treating the remaining instances as pseudo-unlabeled ones. Then we design a SS learning objective, consisting of a supervised loss for pseudo-labeled instances and a semantic consistency regularization for pseudo-unlabeled instances. We further introduce a complementary regularization for those non-candidate labels to constrain the model predictions on them to be as small as possible. Empirical results demonstrate that PLSP significantly outperforms the existing PL baseline methods, especially on high ambiguity levels. Code available: https://github.com/changchunli/PLSP.

Concept relatedness estimation (CRE) aims to determine whether two given concepts are related. Existing methods only consider the pairwise relationship between concepts, while overlooking the higher-order relationship that could be encoded in a concept-level graph structure. We discover that this underlying graph satisfies a set of intrinsic properties of CRE, including reflexivity, commutativity, and transitivity. In this paper, we formalize the CRE properties and introduce a graph structure named ConcreteGraph. To address the data scarcity issue in CRE, we introduce a novel data augmentation approach to sample new concept pairs from the graph. As it is intractable for data augmentation to fully capture the structural information of the ConcreteGraph due to a large amount of potential concept pairs, we further introduce a novel Graph Component Contrastive Learning framework to implicitly learn the complete structure of the ConcreteGraph. Empirical results on three datasets show significant improvement over the state-of-the-art model. Detailed ablation studies demonstrate that our proposed approach can effectively capture the high-order relationship among concepts.

In the past few years, Deep Reinforcement Learning (DRL) has become a valuable solution to automatically learn efficient resource management strategies in complex networks. In many scenarios, the learning task is performed in the Cloud, while experience samples are generated directly by edge nodes or users. Therefore, the learning task involves some data exchange which, in turn, subtracts a certain amount of transmission resources from the system. This creates a friction between the need to speed up convergence towards an effective strategy, which requires the allocation of resources to transmit learning samples, and the need to maximize the amount of resources used for data plane communication, maximizing users' Quality of Service (QoS), which requires the learning process to be efficient, i.e., minimize its overhead. In this paper, we investigate this trade-off and propose a dynamic balancing strategy between the learning and data planes, which allows the centralized learning agent to quickly converge to an efficient resource allocation strategy while minimizing the impact on QoS. Simulation results show that the proposed method outperforms static allocation methods, converging to the optimal policy (i.e., maximum efficacy and minimum overhead of the learning plane) in the long run.