Reviews: Online Robust Policy Learning in the Presence of Unknown Adversaries

Summary: This work considers the problem of learning a robust policy in a scenario where state inputs to the policy are subject to intermittent periods of adversarial attack. The authors propose a meta-learning based approach, whereby separate sub-policies are learned for the nominal and adversarial conditions, and at each time step a master policy selects one of these sub-policies to execute on the basis of their respective advantage estimates for the current observed state. Qualitative assessment: The idea of using advantage estimates to detect adversarial attacks is quite appealing, since by definition an adversarial attack should decrease the advantage of the policy regardless of the attack method used. However, a disadvantage is that the attack can only be detected after a delay, since the agent must first experience an unexpectedly low reward. This seems like it would be especially problematic in domains with long time horizons and sparse rewards, where the consequences of selecting a suboptimal action may not become apparent until much later on.