We propose an adaptive (stochastic) gradient perturbation method for differentially private empirical risk minimization. At each iteration, the random noise added to the gradient is optimally adapted to the stepsize; we name this process adaptive differentially private (ADP) learning. Given the same privacy budget, we prove that the ADP method considerably improves the utility guarantee compared to the standard differentially private method in which vanilla random noise is added. Our method is particularly useful for gradient-based algorithms with time-varying learning rates, including variants of AdaGrad (Duchi et al., 2011). We provide extensive numerical experiments to demonstrate the effectiveness of the proposed adaptive differentially private algorithm.

Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance on a wide variety of tasks. However, their accuracy comes at the cost of intelligibility: it is usually unclear how they make their decisions. This hinders their applicability to high stakes decision-making domains such as healthcare. We propose Neural Additive Models (NAMs) which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models. NAMs learn a linear combination of neural networks that each attend to a single input feature. These networks are trained jointly and can learn arbitrarily complex relationships between their input feature and the output. Our experiments on regression and classification datasets show that NAMs are more accurate than widely used intelligible models such as logistic regression and shallow decision trees. They perform similarly to existing state-of-the-art generalized additive models in accuracy, but are more flexible because they are based on neural nets instead of boosted trees. To demonstrate this, we show how NAMs can be used for multitask learning on synthetic data and on the COMPAS recidivism data due to their composability, and demonstrate that the differentiability of NAMs allows them to train more complex interpretable models for COVID-19.

SambaNova Systems, a Palo Alto–based AI startup, announced a new language service model with a familiar description: GPT, which stands for Generative Pre-trained Transformer, and has no links to OpenAI's GPT series of language models. SambaNova markets their GPT as an everyman's alternative to OpenAI's GPT-3, writing in the press release that it will allow companies "to be up and running with a customized language model in as fast as one month as opposed to nine months or a year." What it's all about: You may have heard of SaaS (Software as a Service) or IaaS (Infrastructure as a Service), but SambaNova Systems offers DaaS: Dataflow-as-a-Service. The aim is to sell a suite of AI tools, including natural language processing ones, for startups to adopt quickly and seamlessly. GPT is the latest tool in its box: a model that can both produce and process natural language. It's built for enterprise use cases, according to the company.

This article is intended for data scientists who may consider using deep learning algorithms, and want to know more about the cons of implementing these type of models into your work. Deep learning algorithms have many benefits, are powerful, and can be fun to show off. However, there are a few times when you should avoid them. I will be discussing those times when you should stop using deep learning below, so keep on reading if you would like a deeper dive into deep learning. Because other algorithms have been around longer, they have countless amounts of documentation, including examples and functions that make interpretability easier.

Artificial neural networks (ANN), usually simply called neural networks, are computing systems inspired by the biological neural networks that constitute animal brains.

OpenAI has recently published an important work, focused on the alignment problem, the problem of ensuring that general-purpose AI and machine learning systems align with human intentions. The "Paperclip Maximizer" is a famous example of alignment gone wrong. To test scalable alignment methods, OpenAI trained a model to summarize entire books, as described in their blog on KDnuggets: Scaling human oversight of AI systems for difficult tasks – OpenAI approach. OpenAI model works by first summarizing small sections of a book, then summarizing those summaries into a higher-level summary, and so on. The results were pretty amazing, so we have asked OpenAI to summarize two top KDnuggets blogs from last year, and here are the summaries.

A new neuroscience study backed with funding from Wellcome and the European Research Council demonstrates how an AI deep learning algorithm is able to predict behavior by decoding brain activity. "The neural code provides a complex, non-linear representation of stimuli, behaviors, and cognitive states," wrote scientists affiliated with the Kavli Institute for Systems Neuroscience, the Max Planck Institute for Human Cognitive and Brain Sciences, UCL, and other institutions in eLife. "Reading this code is one of the primary goals of neuroscience – promising to provide insights into the computations performed by neural circuits." The decoding of brain data from imaging and neural recordings is a complex, time-consuming undertaking that the study's scientists characterize as "a non-trivial problem, requiring strong prior knowledge about the variables encoded and, crucially, the form in which they are represented." In efforts to decipher the neural code, the researchers created a convolutional neural network (CNN) to predict behaviors or other co-recorded stimuli from minimally processed, wide-band neural data.

Artificial intelligence (AI) is a cutting-edge technology that enables robots to learn from their own experience. AI can be found in self-driving cars, smart homes, and chess computers, to name a few. They are based on deep learning and are equipped with artificial intelligence. Computers can execute complex tasks using these technologies. As a result, businesses are recognized for their enthusiasm for AI to obtain a competitive advantage over their competitors.

"Just use AI!" While you may have heard this before, using artificial intelligence can seem like a lot of work. But it doesn't have to be, and there are many AI APIs out there ready for you to leverage. Check out some of them in Postman's latest featured list, Artificial Intelligence APIs, then get started straight away by forking any or all of these popular APIs to your own workspace. OpenAI is a non-profit AI research company whose goal is to advance digital intelligence. They've been widely talked about recently when they announced Codex, an AI that translates natural language to code.

In this paper, we introduce the Layer-Peeled Model, a nonconvex, yet analytically tractable, optimization program, in a quest to better understand deep neural networks that are trained for a sufficiently long time. As the name suggests, this model is derived by isolating the topmost layer from the remainder of the neural network, followed by imposing certain constraints separately on the two parts of the network. We demonstrate that the Layer-Peeled Model, albeit simple, inherits many characteristics of well-trained neural networks, thereby offering an effective tool for explaining and predicting common empirical patterns of deep-learning training. First, when working on class-balanced datasets, we prove that any solution to this model forms a simplex equiangular tight frame, which, in part, explains the recently discovered phenomenon of neural collapse [V. More importantly, when moving to the imbalanced case, our analysis of the Layer-Peeled Model reveals a hitherto-unknown phenomenon that we term Minority Collapse, which fundamentally limits the performance of deep-learning models on the minority classes.