1 2 " Xt Ut # 0 " Hxxt Hxut Huxt Huut

Based onLemma 5.1anditsproof, weknownthatthePMP oftheauxiliary control system, (S.2), is exactly the differential PMP equations (13). Thus below, we only look at the differential PMP equationsin(S.2). In the system identification experiment, we collect a total number of five trajectories from systems (in Table 2) with dynamics known, wherein different trajectoriesξo = {xo0:T,u0:T 1}havedifferent initial conditionsx0 andhorizonsT (T ranges from10to20),with randominputsu0:T 1 drawnfromuniformdistribution. In fact, throughout the entire learning process, PDP always guarantees that the policyconstraint isperfectly respected (as the forward pass strictly follows the policy). Please seeAppendix Fig. S4for validation.