Reviews: Backprop with Approximate Activations for Memory-efficient Network Training

This paper proposes the elegant and obvious-in-retrospect idea of using exact activations for the forward pass and low-precision activations for the backward pass, thereby achieving nearly the full memory savings of low-precision activations. It shows that this scheme nearly matches the exact training curves while allowing 4-bit precision. Overall, the paper is well-executed. The writing is clear, references to related work are pretty complete, and the experiments seem sensible and convincing. The reviewers feel like the paper could have been more ambitious in certain respects (e.g.