A Appendix

Hyper-parameter Setup The pre-training hyper-parameters of Transcormer are described in Table 8. As mentioned in Section 2.1, some works [ MLM model caused by N-passes. K tokens via masked prediction as the final sentence probability. To fulfill this target, DLM only feeds word embeddings as the key/value for each Transformer layer, rather than the previous layer. Just as discussed in Section 3.3, this model learns forward and backward A.3 Results A.3.1 Comparison with other works As aforementioned, previous works [35, 34] have tried some strategies to calculate the probabilities MLM adopts one bidirectional context and SLM adopts forward and backward contexts.