Supplementary Material Learning to Play Sequential Games versus Unknown Opponents Pier Giuseppe Sessa, Ilija Bogunovic, Maryam Kamgarpour, Andreas Krause (NeurIPS 2020)