
Machine learned models are increasingly entering wider ranges ofdomains inour lives, driving a constantly increasing number of important systems. Large scale systems can be trained in highly parallel and distributed training environments, with a large amount of randomness in training the models.