Deep Learning with Intel's BigDL and Apache Spark - Cloudera Engineering Blog

We can also independently test the model performance on a test set using any of the trained model snapshots saved at the checkpoint location. If ever the model performance improves initially and then starts to flatten or decrease it might be a good idea to reduce the learning rate at that point while resuming training from where it left off. All one would need to do is use the model snapshot from the 15th epoch, which would be a minor change to the code above.