r/MachineLearning - [D] Rant/Question: scaled dot-product attention

In "Attention Is All You Need" Vaswani et al. propose to scale the value of the dot-product attention score by 1/sqrt(d) before taking the softmax, where d is the key vector size. Clearly, this scaling should depend on the initial value of the weights that compute the key and query vectors, since the scaling is a reparametrization of these weight matrices, but unfortunately the paper does not specify how these weights are initialized. Trying to follow the rabbit hole that is tensor2tensor, which is supposed to be the reference implementation, it seems to use the default tf.layers.Conv2D initalizer which is undocumented, but people on teh interwebz say that it is Glorot uniform. Marian also uses Glorot uniform, while Sockeye uses the default MXNet initializer which I don't know what it is. Rant: the initializer should have been clearly specified in the paper, or at lest in the reference code.