Using Dropout with Neural Networks: Not A Magic Bullet

Overfitting is an issue that occurs when a model shows high accuracy in predicting training data (the data used to build the model), but low accuracy in predicting test data (unseen data that the model has not used before). This can particularly be a problem when it comes to using small datasets in the course of building a neural network. It is possible for the neural network to be of such a size that it "overtrains" on the training data -- and therefore performs poorly when it comes to predicting new data. This is to prevent excessive "noise" in the network that artificially increases the training accuracy, but does not result in any meaningful information being transferred to the output layer -- i.e. any increase in the training accuracy comes from excessive training and not from any useful information from the model features themselves. Dropout renders certain nodes in the network inactive as illustrated in the image at the beginning of this article -- thus forcing the network to look for more meaningful patterns that influence the output layer.