Appendix details of the proposed method

In this section, we provide further intuition about the proposed AdaQN method. As shown in Figure 5, in AdaQN we need to ensure that the approximate solution of the ERM problem with m samples denoted by wm is within the superlinear convergence neighborhood of BFGS for the ERM problem with n = 2msamples. Here, w m and w n are the optimal solutions of the risks Rm and Rn corresponding to the sets Sm and Sn with mand nsamples, respectively, where Sm Sn. The statistical accuracy region of Rm is denoted by a blue circle, the statistical accuracy region of Rn is denoted by a red circle, and the superlinear convergence neighborhood of BFGS for Rn is denoted by a dotted purple circle. As we observe, any point within the statistical accuracy of w m is within the superlinear convergence neighborhood of BFGS for Rn.