We thank the reviewers for their constructive feedback on the paper. Here we answer their main questions and comments. In addition, are the results shown significant? In particular, we have evaluated our approach for transfer learning for low-resource and/or fine-grained classification. Then (3) we use our method, i.e. we fine-tune the last Finally, we applied our method to a very large ResNeXt-101 32x48d from [Mahajan et al.

Several papers consider approaches that limit the number of bits to represent floating point numbers [13, 24, 31]. Recent work proposes adaptive tuning of the compression ratio [7].

Existing research on video representation for processing has primarily focused on 2D/2.5D

We perform computational experiments that support our theoretical results.

Neural Information Processing Systems http://nips.cc/

These authors contributed equally to this work.

Figure 2: Anexampleofadynamic (left) andstatic (right) neuralgraph. Claim: Assume L is Lipschitzcontinuous.

R2: Sparsity: Thank you for mentioning thatthe unification ofcoreparts ofthe sparseneuralnetwork literature14 combined with the neural architecture search problem is very nice. As a useful experiment to include to showcase15 this, we apply Algorithm 1 separately to each filter of the version of ResNet50 used in [1] and obtain state of the16 art performance atthe task oftraining sparse networks from scratchas described in [1]. R2: Theoretical Claims: We agree that this work will benefit from theoretical analysis which extends beyond25 decreasing the loss on the mini-batch with the SGD update. We thank you for the excellent suggestion of offering the additional motivation of31 performing an approximation in the backwards pass. We are happy to see that we may arrive at the same objec-32 tive in this way and we will include this exciting result.

This is meaningful because in most applications, the time complexities for evaluating gradients at different points are of the same magnitude.