A Supplementary Material A.1 Side-by-side comparison of MDP and tMDPA temporal MDP process: (S, A,p

This proof draws closely to the proof of the temporal policy gradient theorem. We shall now prove that, under Assumption 4.2, the B&B process can be formulated as a Second, Lemma A.1, together with Assumption 4.2, ensures the existence of (deterministic) distributions This concludes the proof.Proposition 4.4. In Depth-First-Search B&B (DFS B&B), that is, when nodes are processed depth-first and left-first by the algorithm, Assumption 4.2 holds. Solid lines show the moving average. The results are averaged over the solving runs that finished successfully for all methods.