To unlock the mystical blackbox of backpropagation, for new machine learning enthusiasts, I've created this short analogy. To use an everyday analogy, we'll consider cooking your favorite food!! To cook your favorite food, you'll need ingredients. To get/buy your ingredients, you'll need money. The amount of money you're willing to spend (budget) influences how much you can spend on your ingredients, and the amount of ingredients you have would determine how many portions of your favorite food that you can prepare.

Neural networks are widely deployed models across many scientific disciplines and commercial endeavors ranging from edge computing and sensing to large-scale signal processing in data centers. The most efficient and well-entrenched method to train such networks is backpropagation, or reverse-mode automatic differentiation. To counter an exponentially increasing energy budget in the artificial intelligence sector, there has been recent interest in analog implementations of neural networks, specifically nanophotonic neural networks for which no analog backpropagation demonstration exists. We design mass-manufacturable silicon photonic neural networks that alternately cascade our custom designed "photonic mesh" accelerator with digitally implemented nonlinearities. These reconfigurable photonic meshes program computationally intensive arbitrary matrix multiplication by setting physical voltages that tune the interference of optically encoded input data propagating through integrated Mach-Zehnder interferometer networks. Here, using our packaged photonic chip, we demonstrate in situ backpropagation for the first time to solve classification tasks and evaluate a new protocol to keep the entire gradient measurement and update of physical device voltages in the analog domain, improving on past theoretical proposals. Our method is made possible by introducing three changes to typical photonic meshes: (1) measurements at optical "grating tap" monitors, (2) bidirectional optical signal propagation automated by fiber switch, and (3) universal generation and readout of optical amplitude and phase. After training, our classification achieves accuracies similar to digital equivalents even in presence of systematic error. Our findings suggest a new training paradigm for photonics-accelerated artificial intelligence based entirely on a physical analog of the popular backpropagation technique.

The success of many neural networks depends on the backpropagation algorithms using which they have been trained. The backpropagation algorithm computes the gradient of the loss function with respect to the weights of a two-layered single input-output network. These algorithms are hardly comparable but we can compare them when we understand their working. Visualization of the procedure of any algorithm is one of the best ways to understand that algorithm. In this article, we will try to understand some of the backpropagation algorithms by visualizing their work.

At the heart of backpropagation are operations and functions which can be elegantly represented as a computational graph. Let's see an example: consider the function f z(x y); It's computational graph representation is shown below: A computational graph is essentially a directed graph with functions and operations as nodes. Computing the outputs from the inputs is called the forward pass, and it's customary to show the forward pass above the edges of the graph. In the backward pass, we compute the gradients of the output wrt the inputs and show them below the edges. Here, we start from the end and go to the beginning computing gradients along the way.

The backpropagation Algorithm is broadly used in machine learning. This algorithm is greatly used for training feed-forward neural networks. It permits the information from the cost to then flow backward through the network, acceptable to compute the gradient. Backpropagation is the core of neural network training. It is the way of adjusting the weights of a neural network.

Backpropagation and gradient descent are two different methods that form a powerful combination in the learning process of neural networks. Let's try to understand the intuition of how this works. Neural networks learn through forward propagation, by using weights, biases, and nonlinear activation functions to calculate a prediction y from the input x that should match the true output y as closely as possible. There are several different loss functions and which one you choose depends on the type of machine learning problem you are facing. The goal of backpropagation is to adjust the weights and biases throughout the neural network based on the calculated cost so that the cost will be lower in the next iteration.

Using backpropagation to compute gradients of objective functions for optimization has remained a mainstay of machine learning. Backpropagation, or reverse-mode differentiation, is a special case within the general family of automatic differentiation algorithms that also includes the forward mode. We present a method to compute gradients based solely on the directional derivative that one can compute exactly and efficiently via the forward mode. We call this formulation the forward gradient, an unbiased estimate of the gradient that can be evaluated in a single forward run of the function, entirely eliminating the need for backpropagation in gradient descent. We demonstrate forward gradient descent in a range of problems, showing substantial savings in computation and enabling training up to twice as fast in some cases.

We propose a novel algorithm called Backpropagation Neural Tree (BNeuralT), which is a stochastic computational dendritic tree. BNeuralT takes random repeated inputs through its leaves and imposes dendritic nonlinearities through its internal connections like a biological dendritic tree would do. Considering the dendritic-tree like plausible biological properties, BNeuralT is a single neuron neural tree model with its internal sub-trees resembling dendritic nonlinearities. BNeuralT algorithm produces an ad hoc neural tree which is trained using a stochastic gradient descent optimizer like gradient descent (GD), momentum GD, Nesterov accelerated GD, Adagrad, RMSprop, or Adam. BNeuralT training has two phases, each computed in a depth-first search manner: the forward pass computes neural tree's output in a post-order traversal, while the error backpropagation during the backward pass is performed recursively in a pre-order traversal. A BNeuralT model can be considered a minimal subset of a neural network (NN), meaning it is a "thinned" NN whose complexity is lower than an ordinary NN. Our algorithm produces high-performing and parsimonious models balancing the complexity with descriptive ability on a wide variety of machine learning problems: classification, regression, and pattern recognition.

In modern neural networks like Transformers, linear layers require significant memory to store activations during backward pass. This study proposes a memory reduction approach to perform backpropagation through linear layers. Since the gradients of linear layers are computed by matrix multiplications, we consider methods for randomized matrix multiplications and demonstrate that they require less memory with a moderate decrease of the test accuracy. Also, we investigate the variance of the gradient estimate induced by the randomized matrix multiplication. We compare this variance with the variance coming from gradient estimation based on the batch of samples. We demonstrate the benefits of the proposed method on the fine-tuning of the pre-trained RoBERTa model on GLUE tasks.

Highlights: In Machine Learning, a backpropagation algorithm is used to compute the loss for a particular model. The most common starting point is to use the techniques of single-variable calculus and understand how backpropagation works. However, the real challenge is when the inputs are not scalars but of matrices or tensors. In this post [1], we will learn how to deal with inputs like vectors, matrices, and tensors of higher ranks. We will understand how backpropagation with vectors and tensors is performed in computational graphs using single-variable as well as multi-variable derivatives.