LM-Switch: Lightweight Language Model Conditioning in Word Embedding Space