Reviews: Policy Optimization via Importance Sampling

Summary The authors present a reinforcement learning technique based on importance sampling. A theoretical analysis is performed that shows how the importance sampling approach affect the upper bound of the expected performance of the target policy using samples from a behavioral policy. The authors propose a surrogate objective function that explicitly mitigates the variance of the policy update due to IS. Two algorithms are proposed based on natural gradients for control-based (learning low-level policy) and parameter-based problems (discrete low-level controller with stochastic upper-level policy). The algorithms were tested on standard control tasks and are compared to state of the art methods.