Attention in Deep Learning

What is the problem we are trying to solve with attention? There is a bottleneck at the end of encoder RNN. Gets worse the longer the sentence. Get dot product of encoder hidden states and first decoder hidden state (green). The more aligned (similar), the bigger the dot product.