Understanding Recurrent Neural Networks (RNNs) from Scratch

Humans do not reboot their understanding of language each time we hear a sentence. Given an article, we grasp the context based on our previous understanding of those words. One of the defining characteristics we possess is our memory (or retention power). Can an algorithm replicate this? The first technique that comes to mind is a neural network (NN).