A Quick Guide to Cross-Entropy Loss Function

One of the most prominent tasks at which Machine Learning has been historically very good at is classifying items (e.g. Especially in recent years, advancements in the hardware capable of executing mathematical models such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs, LSTMs) made it possible to perform a quantum leap (sometimes literally) in performance. However, defining a model is only half of the story. To find the best parameters to perform such a task, it is also necessary to define a cost or loss function that captures the essence of what we want to optimize, and execute some form of gradient descent to reach a suitable set of parameters. To begin with, we need to define a statistical framework that describes our problem.