A complete understanding of neural network mathematics for backpropagation in more complex networks requires an understanding of more esoteric multivariable calculus notions, like the Jacobian. I'll post an article in the future on this once I have a better grasp of the concept itself. But, we can achieve quite a lot of intuition at a slightly more basic level. Turns out that we already did a bulk of the work -- computing backpropagation on a larger neural network just requires a few logical steps. Ignoring the biases, let's see if we can use similar logic to backpropagate through are more complex network, this time to two different weights.

And since we now have probabilities we can calculate the Cross Entropy as we have reviewed earlier.

Loss Functions are one of the most important parts of Neural Network design. A loss function helps us to interact with the model and tell the model what we want -- the reason why it is related to an "objective function". Let us look at the precise definition of a loss function. In mathematical optimization and decision theory, a loss function or cost function is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cost" associated with the event. An optimization problem seeks to minimize a loss function.