From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function