Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space