Explaining Machine Learning Models for Natural Language

We also investigate one popular method for faithfully explaining neural NLP models: attention weights. Attention is a highly successful model architecture component used in machine learning. It produces an intermediate set of values called weights, which correspond to locations of individual words in a given input: a higher attention weight means that (a vector artifact corresponding to the location of) that word was more important for the model's prediction. For example, if the goal is to predict the label "five stars" for a movie review, the model might learn to place larger attention weights on the vectors representing words "loved" and "good" rather than "the", "and", or "movie", which don't really indicate much about the author's feelings towards the film. See Figure 1 for an example of attention weights on a movie review.