Notes for deep learning on NLP

Deep learning gradually plays a major role on NLP (Natural Language Processing). Here I note some technical evolution for the NLP problems. A continuous text sequence "to be or not to be" can be modelled by: N-gram model can solve the problem of next word prediction, e.g., the occurrence of 6-gram model can predict the probability of next word is "be" if the previous words are "to be or not to": With neural network, the idea is proposed to train a shared matrix C which can project each word into a feature vector, and put the vector as the input of a neural network to train the main task. Suppose the dimension of feature space is M, and vocabluary is V, the projection C is a V *M matrix. The input layer contains N-1 previous words in a N-gram model, which is encoded by 1-to- V representation.