Artificial neural networks can achieve impressive performances, and even outperform humans in some specific tasks. Nevertheless, unlike biological brains, the artificial neural networks suffer from tiny perturbations in sensory input, under various kinds of adversarial attacks. It is therefore necessary to study the origin of the adversarial vulnerability. Here, we establish a fundamental relationship between geometry of hidden representations (manifold perspective) and the generalization capability of the deep networks. For this purpose, we choose a deep neural network trained by local errors, and then analyze emergent properties of trained networks through the manifold dimensionality, manifold smoothness, and the generalization capability. To explore effects of adversarial examples, we consider independent Gaussian noise attacks and fast-gradient-sign-method (FGSM) attacks. Our study reveals that a high generalization accuracy requires a relatively fast power-law decay of the eigen-spectrum of hidden representations. Under Gaussian attacks, the relationship between generalization accuracy and power-law exponent is monotonic, while a non-monotonic behavior is observed for FGSM attacks. Our empirical study provides a route towards a final mechanistic interpretation of adversarial vulnerability under adversarial attacks.

Over-parameterization is ubiquitous nowadays in training neural networks to benefit both optimization in seeking global optima and generalization in reducing prediction error. However, compressive networks are desired in many real world applications and direct training of small networks may be trapped in local optima. In this paper, instead of pruning or distilling over-parameterized models to compressive ones, we propose a new approach based on differential inclusions of inverse scale spaces. Specifically, it generates a family of models from simple to complex ones that couples a pair of parameters to simultaneously train over-parameterized deep models and structural sparsity on weights of fully connected and convolutional layers. Such a differential inclusion scheme has a simple discretization, proposed as Deep structurally splitting Linearized Bregman Iteration (DessiLBI), whose global convergence analysis in deep learning is established that from any initializations, algorithmic iterations converge to a critical point of empirical risks. Experimental evidence shows that DessiLBI achieve comparable and even better performance than the competitive optimizers in exploring the structural sparsity of several widely used backbones on the benchmark datasets. Remarkably, with early stopping, DessiLBI unveils "winning tickets" in early epochs: the effective sparse structure with comparable test accuracy to fully trained over-parameterized models.

Sparse neural networks are effective approaches to reduce the resource requirements for the deployment of deep neural networks. Recently, the concept of adaptive sparse connectivity, has emerged to allow training sparse neural networks from scratch by optimizing the sparse structure during training. However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions. This comparison becomes increasingly complex as the number of possible topological comparisons increases exponentially with the size of networks. In this work, we introduce an approach to understand and compare sparse neural network topologies from the perspective of graph theory. We first propose Neural Network Sparse Topology Distance (NNSTD) to measure the distance between different sparse neural networks. Further, we demonstrate that sparse neural networks can outperform over-parameterized models in terms of performance, even without any further structure optimization. To the end, we also show that adaptive sparse connectivity can always unveil a plenitude of sparse sub-networks with very different topologies which outperform the dense model, by quantifying and comparing their topological evolutionary processes. The latter findings complement the Lottery Ticket Hypothesis by showing that there is a much more efficient and robust way to find "winning tickets". Altogether, our results start enabling a better theoretical understanding of sparse neural networks, and demonstrate the utility of using graph theory to analyze them.

The COVID crisis has skyrocketed the applications of artificial intelligence -- from tackling this global pandemic, to being a vital tool in managing various business processes. Despite its benefits, AI has always been scrutinised for its ethical concerns like existing biases and privacy issues. However, this technology also has some significant sustainability issues – it is known to consume a massive amount of energy, creating a negative impact on the environment. As AI technology is getting advanced in predicting weather, understanding human speech, enhancing banking payments, and revolutionising healthcare, the advanced models are not only required to be trained on large datasets, but also require massive computing power to improve its accuracy. Such heavy computing and processing consumes a tremendous amount of energy and emits carbon dioxide, which has become an environmental concern. According to a report, it has been estimated that the power required for training AI models emits approximately 626,000 pounds (284 tonnes) of carbon dioxide, which is comparatively five times the lifetime emissions of the average US car.

Artificial intelligence uses the techniques such as natural language processing, machine learning, adaptive learning, deep learning, and computer vision …

Supervised, semi-supervised or unsupervised deep learning is part of a broader family of machine learning methods, that teach you the basics of neural networks. Learn from the Top 10 Deep Learning Courses curated exclusively by Analytics Insight and build your deep learning models with Python and NumPy. Taught by one of the best Data Science experts of 2020 Andrew Ng, this course teaches you how to build a successful machine learning project. You will understand the complex ML settings, such as mismatched training/test sets, and comparing to and/or surpassing human-level performance. Over 20 videos spread across the entire module will explain you error analysis and different kind of the learning techniques.

I had an interview with a big company and was asked a number of questions, but one that stuck out to me was when the interviewer asked "Do…

The perceptron is the most basic of all neural networks, being a fundamental building block of more complex neural networks. It simply connects an input cell and an output cell. The feed-forward network is a collection of perceptrons, in which there are three fundamental types of layers -- input layers, hidden layers, and output layers. During each connection, the signal from the previous layer is multiplied by a weight, added to a bias, and passed through an activation function. Feed-forward networks use backpropagation to iteratively update the parameters until it achieves a desirable performance.

Machine learning solutions in the real world are rarely just a matter of building and testing models. Managing and automating the lifecycle of machine learning models from training to optimization is, by far, the hardest problem to solve in machine learning solutions. To control the lifecycle of a model, data scientists need to be able to persist and query its state at scale. This problem might seem trivial until you consider that any average deep learning model can include hundreds of hidden layers and millions of interconnected nodes;) Storing and accessing large computation graphs is far from trivial. Most of the times, data science teams spend a lot of time trying to adapt commodity NOSQL databases to machine learning models before arriving to the not-so-obvious conclusion: Machine learning solutions need a new type of database.

Developers generally exhibit a strong affinity (usually paired with an equally strong hatred) for certain frameworks, libraries, and tools. But which ones do they love, dread, and want the most? Stack Overflow, as part of its enormous, annual Developers Survey, asked that very question, and the answers provide some interesting insights into how developers work. Some 65,000 developers responded to the survey, and the sheer size of that sample makes these breakdowns a bit more interesting to parse. For example, although game developers might have strong opinions about Unreal Engine and Unity 3D (which placed high on the following lists), those aren't used at all by the bulk of developers concerned with A.I. and machine learning, who have strong feelings about TensorFlow that many other developers might not share.