Paying Attention Again!

The Transformer Architecture [1] introduced by Vaswani et al, is based on attention mechanism and overcomes the challenges faced in recurrence. In continuation to the last blog'Let's pay some Attention!', Let's do a quick recap of the attention mechanism we understood in the last blog. We have some key - value pairs and a query. We compare the query to each key and the key with the highest similarity score is assigned the highest weight.