Language Model Re-rankers are Steered by Lexical Similarities