Peeking into the neural network architecture used for Google's Neural Machine Translation

The Google Neural Machine Translation paper (GNMT) describes an interesting approach towards deep learning in production. The paper and architecture are non-standard, in many cases deviating far from what you might expect from an architecture you'd find in an academic paper. Emphasis is placed on ensuring the system remains practical rather than chasing the state of the art through typical but computationally intensive tweaks. To understand the model used in GNMT we'll start with a traditional encoder decoder machine translation model and keep evolving it until it matches GNMT. The GNMT evolution seems primarily motivated by improving accuracy while maintaining practical production speeds for both training and prediction. The encoder decoder architecture started the recent neural machine translation trend.