Why is Your Language Model a Poor Implicit Reward Model?