Supplement to " Sample-Efficient Reinforcement Learning for Linearly-Parameterized MDPs with a Generative Model " Bingyan Wang