Goto

Collaborating Authors

Results


A Tutorial on Spiking Neural Networks for Beginners

#artificialintelligence

Despite being quite effective in a variety of tasks across industries, deep learning is constantly evolving, proposing new neural network (NN) architectures, deep learning (DL) tasks, and even brand new concepts of the next generation of NNs, such as the Spiking Neural Network (SNN). SNN was introduced by the researchers at Heidelberg University and the University of Bern developing as a fast and energy-efficient technique for computing using spiking neuromorphic substrates. In this article, we will mostly discuss Spiking Neural Network as a variant of neural network. We will also try to understand how is it different from the traditional neural networks. Below is a list of the important topics to be tackled.


Under the Hood of Modern Machine and Deep Learning

#artificialintelligence

In this chapter, we investigate whether unique, optimal decision boundaries can be found. In order to do so, we first have to revisit several fundamental mathematical principles. Regularization is a mathematical tool, which allows us to find unique solutions even for highly ill-posed problems. In order to use this trick, we review norms and how they can be used to steer regression problems. Rosenblatt's Perceptron and Multi-Layer Perceptrons which are also called Artificial Neural Networks inherently suffer from this ill-posedness.


Under the Hood of Modern Machine and Deep Learning

#artificialintelligence

In this chapter, we investigate whether unique, optimal decision boundaries can be found. In order to do so, we first have to revisit several fundamental mathematical principles. Regularization is a mathematical tool, which allows us to find unique solutions even for highly ill-posed problems. In order to use this trick, we review norms and how they can be used to steer regression problems. Rosenblatt's Perceptron and Multi-Layer Perceptrons which are also called Artificial Neural Networks inherently suffer from this ill-posedness.


Robustness of Neural Networks to Parameter Quantization

arXiv.org Machine Learning

Quantization, a commonly used technique to reduce the memory footprint of a neural network for edge computing, entails reducing the precision of the floating-point representation used for the parameters of the network. The impact of such rounding-off errors on the overall performance of the neural network is estimated using testing, which is not exhaustive and thus cannot be used to guarantee the safety of the model. We present a framework based on Satisfiability Modulo Theory (SMT) solvers to quantify the robustness of neural networks to parameter perturbation. To this end, we introduce notions of local and global robustness that capture the deviation in the confidence of class assignments due to parameter quantization. The robustness notions are then cast as instances of SMT problems and solved automatically using solvers, such as dReal. We demonstrate our framework on two simple Multi-Layer Perceptrons (MLP) that perform binary classification on a two-dimensional input. In addition to quantifying the robustness, we also show that Rectified Linear Unit activation results in higher robustness than linear activations for our MLPs.


Artificial Neural Networks

arXiv.org Machine Learning

The term neural networks refers to networks of neurons in the mammalian brain. Neurons are its fundamental units of computation. In the brain they are connected together in networks to process data. This can be a very complex task, and the dynamics of neural networks in the mammalian brain in response to external stimuli can therefore be quite intricate. Inputs and outputs of each neuron vary as functions of time, in the form of so-called spike trains, but also the network itself changes. We learn and improve our data-processing capacities by establishing reconnections between neurons. Neural-networkalgorithms are inspired by the architecture and the dynamics of networks of neurons in the brain. Yet the algorithms use neuron models that are highly simplified, compared with real neurons. Nevertheless, the fundamental principle is the same: artificial neural networks learn by reconnection.


How to Fix Vanishing Gradients Using the Rectified Linear Activation Function

#artificialintelligence

The vanishing gradients problem is one example of unstable behavior that you may encounter when training a deep neural network. It describes the situation where a deep multilayer feed-forward network or a recurrent neural network is unable to propagate useful gradient information from the output end of the model back to the layers near the input end of the model. The result is the general inability of models with many layers to learn on a given dataset or to prematurely converge to a poor solution. Many fixes and workarounds have been proposed and investigated, such as alternate weight initialization schemes, unsupervised pre-training, layer-wise training, and variations on gradient descent. Perhaps the most common change is the use of the rectified linear activation function that has become the new default, instead of the hyperbolic tangent activation function that was the default through the late 1990s and 2000s. In this tutorial, you will discover how to diagnose a vanishing gradient problem when training a neural network model and how to fix it using an alternate activation function and weight initialization scheme.


A Primer on Neural Network Models for Natural Language Processing

Journal of Artificial Intelligence Research

Over the past few years, neural networks have re-emerged as powerful machine-learning models, yielding state-of-the-art results in fields such as image recognition and speech processing. More recently, neural network models started to be applied also to textual natural language signals, again with very promising results. This tutorial surveys neural network models from the perspective of natural language processing research, in an attempt to bring natural-language researchers up to speed with the neural techniques. The tutorial covers input encoding for natural language tasks, feed-forward networks, convolutional networks, recurrent networks and recursive networks, as well as the computation graph abstraction for automatic gradient computation.