Finding the Narrative with Natural Language Processing

In general, text pre-processing should include lowercasing all words, removing punctuation and stop words, and stemming or lemmatization. When working with tweets, in addition to the normal text-preprocessing tasks we also have to consider hashtags, acronyms, re-tweet syntax ('RT @scrapfishies:…'), emojis, and other elements. Should hashtags be be segmented (divided into their unique words) or kept as a single concatenated string? Well, I'd argue that it depends on the hashtag. As an example, the #blacklivesmatter hashtag was used frequently in this corpus -- segmenting would give us 3 distinct tokens: 'black', 'lives', and'matter'.