The study of the generalization of 2

We thank the reviewers for their constructive and positive comments. They will improve the quality of the paper. As an instance in RL, we mention the problem of "active exploration in MDPs" (see [28]), where the Reiterating the discussion in Section 2.3, let us consider the small-budget regime, and We will provide a footnote in page 7 to clarify this. This is indeed a nice remark. As a result, the theorem is valid even if irreducibility and aperiodicity are dropped.