Relative Positional Encoding

In this post, we will take a look at relative positional encoding, as introduced in Shaw et al (2018) and refined by Huang et al (2018). This is a topic I meant to explore earlier, but only recently was I able to really force myself to dive into this concept as I started reading about music generation with NLP language models. This is a separate topic for another post of its own, so let's not get distracted. Let's dive right into it! If you're already familiar with transformers, you probably know that transformers process inputs in parallel at once.