An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutines