Convolutional Neural Networks for Text Categorization: Shallow Word-level vs. Deep Character-level

Text categorization is the task of labeling documents, which has many important applications such as sentiment analysis and topic categorization. Recently, several variations of convolutional neural networks (CNNs) [7] have been shown to achieve high accuracy on text categorization (see e.g., [3, 4, 9, 1] and references therein) in comparison with a number of methods including linear methods, which had long been the state of the art. Long-Short Term Memory networks (LSTMs) [2] have also been shown to perform well on this task, rivaling or sometimes exceeding CNNs [5, 8]. However, CNNs are particularly attractive since, due to their simplicity and parallel processing-friendly nature, training and testing of CNNs can be made much faster than LSTM to achieve similar accuracy [5], and therefore CNNs have a potential to scale better to large training data. Here we focus on two CNN studies that report high performances on categorizing long documents (as opposed to categorizing individual sentences): - Our earlier work (2015) [3, 4]: shallow word-level CNNs (taking sequences of words as input), which we abbreviate as word-CNN.