Explaining Black-Box Machine Learning Models - Code Part 2: Text classification with LIME

Okay, our model above works but there are still common words and stop words in our model that LIME picks up on. Ideally, we would want to remove them before modeling and keep only relevant words. This we can accomplish by using additional steps and options in our preprocessing function. Important to know is that whatever preprocessing we do with our text corpus, train and test data has to have the same features (i.e. If we were to incorporate all the steps shown below into one function and call it separately on train and test data, we would end up with different words in our dtm and the predict() function won't work any more.