Appendix A Continuous RL: Formulation and Well-Posedness 467 A.1 Exploratory Stochastic-Control

Assumption 2. The following conditions are assumed throughout: A; (32) (iv) r has polynomial growth in x and a, i.e., there exists a constant C > 0 and µ 1 such that To do so, let's assume Theorem 6. Assume that for a policy π and for every x, Assumption 3. Assume the following conditions hold: Lemma 9. Let π, ˆ π be two feedback policies. We need a lemma for the perturbation bounds. Here we present a detailed version of the CPPO algorithm. D.3 below, which clearly illustrates the advantage of square-root KL divergence.