Overview of Weight Initialization

Weights can be thought of as the amount of influence the input has on the output. With weight initialization, we set the weights to random values to prevent the layer outputs or gradients from vanishing or exploding. To understand what vanishing and exploding gradients are, let's break down what a neural network does. In a neural network, many matrix multiplications are performed during forward propagation. In forward propagation, inputs are passed through layers to end up with a final output from the output node.