The emergent algebraic structure of RNNs and embeddings in NLP

Tremendous advances in natural language processing (NLP) have been enabled by novel deep neural network architectures and word embeddings. Historically, convolutional neural network (CNN)[1, 2] and recurrent neural network (RNN)[3, 4] topologies have competed to provide state-of-the-art results for NLP tasks, ranging from text classification to reading comprehension. CNNs identify and aggregate patterns with increasing feature sizes, reflecting our common practice of identifying patterns, literal or idiomatic, for understanding language; they are thus adept at tasks involving key phrase identification. RNNs instead construct a representation of sentences by successively updating their understanding of the sentence as they read new words, appealing to the formally sequential and rule-based construction of language. While both networks display great efficacy at certain tasks [5], RNNs tend to be the more versatile, have emerged as the clear victor in, e.g., language translation [6, 7, 8], and are typically more capable of identifying important contextual points through attention mechanisms for, e.g., reading comprehension [9, 10, 11, 12].