Identify, version control, and document the best performing model during training

Model training can be seen as the generation of subsequent versions of a model -- after each batch, the model weights are adjusted, and as a result, a new version of the model is created. Each new version will have varying levels of performance (as evaluated against a validation set). If everything goes well, training and validation loss will decrease with the number of training epochs. However, the best performing version of a model (here abbreviated as best model) is rarely the one obtained at the end of the training process. Take a typical overfitting case -- at first, both training and validation losses decrease as training progresses.