Why doesn't extra supervision increase the performance of the SOTA language model? • /r/MachineLearning

I took the tensorflow implementation of the language model from Zaremba et al., 2014, and changed the loss function from what it was (crossentropy with 1-hot vector representing the correct word) to a loss made up of two terms, the first is the loss from before and the second is a crossentropy loss with a 1-hot vector representing the closest synonym to the target word. I tried playing around with the weighting of these two terms, but no matter what I did the results did not improve over the original model. Doesn't this new loss function basically tell the network "the next correct word is'dog', but if you say its'puppy' thats also OK"?