A Appendix

A.1 Proof of Equation 9 We prove Eq. 9 by contradiction. Due to the observation (2) in Section 4.1, the following inequalities hold: X Algorithm 2 is our proposed algorithm for latency-aware mask search, which extends Algorithm 1. Based on Eq. 5 and the warm-start constraint in Section 4.2, the optimization problem Eq. 4 is written If there exists a mask m that strictly better optimizes Eq. 22 than ˆ m: Although Eq. 27 has a closed form B, we use the numerical solver in CuPy for higher stability. SQuAD 2.0 is an extension of SQuAD 1.1 by including unanswerable The FLOPs constraint is 60 % . There is a trade-off between sample dataset size and accuracy.