SupplementaryMaterialfor BAIL: Best-ActionImitationLearningfor BatchDeepReinforcementLearning

Note that ˆφ is feasible for the constrained optimization problem. We refer to it as an "early stopping scheme" because the key idea is to return to the parameter values which gave the lowest validation error (see Section 7.8 of Goodfellow et al.[3]). In our implementation, we initialize two upper envelope networks with parametersφ and φ0, where φ is trained using the penalty loss, andφ0 records the parameters with the lowest validation error encounteredsofar. IfLφ > Lφ0, we count the number of consecutive times this occurs. Notonlyis this not standard practice, but to makeafair comparison across all algorithms, this would require, foreachofthe fivealgorithms, performing aseparate hyper-parameter search foreachofthe five environments.