Reviews: Reinforcement Learning under Model Mismatch

The paper tackles the robust MDP setting, where the problem being solved lies within some set of possibilities, and the goal is to obtain a policy that does well in the worst case. In particular, the paper starts from the (known) robust Bellman equation and derives a number of model-free algorithms (analogs to Q-learning, SARSA, TD-learning, LSTD, GTD, and more, many with convergence guarantees in the robust MDP setting. The paper itself contains the results of a single experiment with robust Q-learning, with more in the supplemental materials. I cannot say that I have evaluated it all in full detail. That said, it does seem to me that the principle ideas that underly the new derivations are sensible and the conclusions seem reasonable.