A Generalized Framework for Population Based Training

Previous PBT implementations have been synchronized glass-box systems. We propose a general, black-box PBT framework that distributes many asynchronous "trials" (a small number of training steps with warm-starting) across a cluster, coordinated by the PBT controller. The black-box design does not make assumptions on model architectures, loss functions Figure 1: Black-box Service for Population Based Training or training procedures. Our system supports dynamic hyperparameter based on a Worker-Controller framework. Each solid blue schedules to optimize both differentiable and non-differentiable circle represents a training trial. A black arrow represents a metrics. We apply our system to train a state-of-the-art WaveNet trial dependency (usually for warm-starting the model from generative model for human voice synthesis. We show that our PBT a parent's checkpoint) and a gray arrow represents an unselected system achieves better accuracy and faster convergence compared parent trial which loses in a tournament and fails to existing methods, given the same computational resource.