Reinforcement learning (RL) studies the problem where an agent interacts with an unknown environment to optimize cumulative rewards/losses [Sutton and Barto, 2018].

With the success of deep learning, there are growing concerns over interpretability (Lipton, 2018). Ideally, the explanation should be both faithful (reflecting the model's actual behavior) and plausible