A Proofs 438 We first redefine notation for clarity and then provide the proofs of the results in the main paper

We first redefine notation for clarity and then provide the proofs of the results in the main paper. Now we first prove that the iteration in Eq.2 has a fixed point. Proof of Lemma 3.1: Let We present the bound on using empirical Bellman operator compared to the true Bellman operator. The proof can be found in [6]. Proof of Theorem 3.4: Recall that the expression of the V -function iterate is given by: Proof of Theorem 3.6: The proof of this statement is divided into two parts.