A Proof of Theorem 4.1

In this section, we shall provide the proof for Theorem 4.1. Appendix A.1 provides additional useful notations and definitions including but not limited to CMDPs, value function, and distance metrics. Appendix A.2 introduces further assumptions, and A.3 Then, we provide the proof pipeline of Theorem 4.1 A.1 Additional notation and definitions used in the proof Before starting, let's introduce some additional notations useful throughout the theoretical analysis. Throughout the proof, we shall focus on the set of CMDPs introduced in Assumption 4.1. Besides the key properties of the target CMDPs introduced in Assumption 4.1, we shall introduce It is interesting to extend our main Theorem 4.1 to more general cases Finally, we describe the following useful lemma which is essential in proving the main part of Theorem 4.1 when the starting state is not in the With above preliminaries in hand, we are ready to embark on the proof for Theorem 4.1, which is Then we consider the terms of interest in two cases.