Refining Source Representations with Relation Networks for Neural Machine Translation

Although neural machine translation (NMT) with the encoder-decoder framework has achieved great success in recent times, it still suffers from some drawbacks: RNNs tend to forget old information which is often useful in the current step and the encoder only operates over words without considering word relationship. To solve these problems, we introduce relation networks (RNs) to learn better representations of the source. In our method RNs are used to associate source words with each other so that the source representation can memorize all the source words and also contain the relationship between them. Then the source representations and all the relations are fed into the attention component together while decoding, with the main encoder-decoder architecture unchanged. Experiments on several data sets show that our method can improve the translation performance significantly over the conventional encoder-decoder model, and can even outperform the approach involving supervised syntactic knowledge.