The goal of compressed sensing is to estimate a high dimensional vector from an underdetermined system of noisy linear equations. In analogy to classical compressed sensing, here we assume a generative model as a prior, that is, we assume the vector is represented by a deep generative model $G: \mathbb{R}^k \rightarrow \mathbb{R}^n$. Classical recovery approaches such as empirical risk minimization (ERM) are guaranteed to succeed when the measurement matrix is sub-Gaussian. However, when the measurement matrix and measurements are heavy-tailed or have outliers, recovery may fail dramatically. In this paper we propose an algorithm inspired by the Median-of-Means (MOM). Our algorithm guarantees recovery for heavy-tailed data, even in the presence of outliers. Theoretically, our results show our novel MOM-based algorithm enjoys the same sample complexity guarantees as ERM under sub-Gaussian assumptions. Our experiments validate both aspects of our claims: other algorithms are indeed fragile and fail under heavy-tailed and/or corrupted data, while our approach exhibits the predicted robustness.

Overfitting describes the phenomenon that a machine learning model fits the given data instead of learning the underlying distribution. Existing approaches are computationally expensive, require large amounts of labeled data, consider overfitting global phenomenon, and often compute a single measurement. Instead, we propose a local measurement around a small number of unlabeled test points to obtain features of overfitting. Our extensive evaluation shows that the measure can reflect the model's different fit of training and test data, identify changes of the fit during training, and even suggest different fit among classes. We further apply our method to verify if backdoors rely on overfitting, a common claim in security of deep learning. Instead, we find that backdoors rely on underfitting. Our findings also provide evidence that even unbackdoored neural networks contain patterns similar to backdoors that are reliably classified as one class.

Pretrained language models such as BERT, GPT have shown great effectiveness in language understanding. The auxiliary predictive tasks in existing pretraining approaches are mostly defined on tokens, thus may not be able to capture sentence-level semantics very well. To address this issue, we propose CERT: Contrastive self-supervised Encoder Representations from Transformers, which pretrains language representation models using contrastive self-supervised learning at the sentence level. CERT creates augmentations of original sentences using back-translation. Then it finetunes a pretrained language encoder (e.g., BERT) by predicting whether two augmented sentences originate from the same sentence. CERT is simple to use and can be flexibly plugged into any pretraining-finetuning NLP pipeline. We evaluate CERT on 11 natural language understanding tasks in the GLUE benchmark where CERT outperforms BERT on 7 tasks, achieves the same performance as BERT on 2 tasks, and performs worse than BERT on 2 tasks. On the averaged score of the 11 tasks, CERT outperforms BERT. The data and code are available at https://github.com/UCSD-AI4H/CERT

International Conference on Learning Representations (ICLR) and Consultative Group on International Agricultural Research (CGIAR) jointly conducted a challenge where over 800 data scientists globally competed to detect diseases in crops based on close shot pictures. The objective of this challenge is to build a machine learning algorithm to correctly classify if a plant is healthy, has stem rust, or has leaf rust. Wheat rust is a devastating plant disease affecting many crops, reducing yields and affecting the livelihoods of farmers and decreasing food security across Africa. The disease is difficult to monitor at a large scale, making it difficult to control and eradicate. An accurate image recognition model that can detect wheat rust from any image will enable a crowd-sourced approach to monitor crops. The imagery data came from a variety of sources.

Deep Learning and Machine Learning are one of the hottest tech fields to be in right now! The field is exploding with opportunities and career prospects. Machine/Deep Learning techniques are widely used in several sectors nowadays such as banking, healthcare, transportation and technology. Machine learning is the study of algorithms that teach computers to learn from experience. Through experience (i.e.: more training data), computers can continuously improve their performance. Deep Learning is a subset of Machine learning that utilizes multi-layer Artificial Neural Networks. Deep Learning is inspired by the human brain and mimics the operation of biological neurons. A hierarchical, deep artificial neural network is formed by connecting multiple artificial neurons in a layered fashion. The more hidden layers added to the network, the more

A few days ago, OpenAI announced a new successor to their Language Model (LM) - GPT-3. This is the largest model trained so far, with 175 billion parameters. While training this large model has its merits, reading a large portion of 72 pages can be tiresome. In this blog post I'll highlight the parts that I find interesting for people familiar with LMs, who merely wish to know (most of) the important points of this work. "The diversity of tasks the model is able to perform in a zero-shot setting suggests that high-capacity models trained to maximize the likelihood of a sufficiently varied text corpus begin to learn how to perform a surprising amount of tasks without the need for explicit supervision" This is an excerpt from the paper accompanying GPT-2.

Since the dawn of Sci-Fi cinema, society has been fascinated with Artificial Intelligence. Whenever we hear the term "AI", our first thought is typically one of a futuristic robot from movies such as Terminator, The Matrix and I, Robot. Although we might still be a few years away from robots that can think for themselves, there have been significant developments in the fields of machine learning and natural language understanding over the past few years. Applications such as Personal Assistants (Siri/Alexa), chatbots and Question-Answering bots are truly revolutionizing the way we interface with machines and go about our daily lives. Natural Language Understanding (NLU) and Natural Language Generation (NLG) are among the fastest growing applications of AI due to the increasing need to understand and derive meaning from language, with its numerous ambiguities and varied structure. According to Gartner, "By 2019, natural-language generation will be a standard feature of 90 percent of modern BI and Analytics platforms".

The paper focuses on what two different types of Recurrent Neural Networks, namely a recurrent Long Short-Term Memory and a recurrent variant of self-organizing memories, a Temporal Self-Organizing Map, can tell us about speakers' learning and processing a set of fully inflected verb forms selected from the top-frequency paradigms of Italian and German. Both architectures, due to the re-entrant layer of temporal connectivity, can develop a strong sensitivity to sequential patterns that are highly attested in the training data. The main goal is to evaluate learning and processing dynamics of verb inflection data in the two neural networks by focusing on the effects of morphological structure on word production and word recognition, as well as on word generalization for untrained verb forms. For both models, results show that production and recognition, as well as generalization, are facilitated for verb forms in regular paradigms. However, the two models are differently influenced by structural effects, with the Temporal Self-Organizing Map more prone to adaptively find a balance between processing issues of learnability and generalization, on the one side, and discriminability on the other side.

In the race to enable manufacturing plants to increase production in the face of an intermittent human workforce, manufacturers are looking at how to supplement their cameras with AI to give human inspectors the ability to spot defective products immediately and correct the problem. While machine vision has been around for more than 60 years, the recent surge in the popularity of deep learning has elevated this sometimes misunderstood technology to the attention of major manufacturers globally. As CEO of a deep learning software company, I've seen how deep learning is a natural next step from machine vision, and has the potential to drive innovation for manufacturers. How does deep learning differ from machine vision, and how can manufacturers leverage this natural evolution of camera technology to cope with real-world demands? In the 1960s, several groups of scientists, many of them in the Boston area, set forth to solve "the machine vision problem."

Embeddings are one of today's main concepts in machine learning. Learn how to visualize, train and apply embeddings for text and image data. Embeddings are one of today's main concepts in machine learning. They capture the semantics of all kinds of data, such as texts, images and videos. Image classification and sentiment analysis are only two out of many applications.