Accelerating the Computation of UCB and Related Indices for Reinforcement Learning