Harnessing Density Ratios for Online Reinforcement Learning