In the main content, we also report the inference latency of different models in Table 1. We list the statistics of datasets utilized in the neural machine translation tasks in Table 5. The underlined words indicate the masked words in the next iteration. While preprocessing, we use the same vocabulary of BERT models to decode the dataset.

Each component in the framework can be considered as a plug-in unit, making the framework flexible and task agnostic.

Later, we perform max-margin (MM) learning to better distinguish between higher-scored sentences and other high-probability but sub-optimal sentences.

Neural models for text generation require a softmax layer with proper word em-beddings during the decoding phase.

Finally, we describe the training setup for our back-translation experiments. We continue to differentiate our method from other existing works. Our method does not train multiple peer models with EM training either. In each round, a forward (or backward) model takes turn to play the "back-translation" role to train The role is switched in the next round. In other words, source and target are identical.

Our method is applicable to all NMT models.

Our method retains the benefits of per-parameter adaptivity while allowing significantly larger models and batch sizes.