Reviews: Efficient Rematerialization for Deep Networks

Though the paper is clearly written and presents an interesting approach for large model training. It also allows a GPU to train models that cannot fit into the GPU on-device memory. However, it completely ignores how fast a model can be trained after rematerialization. Given that it already takes a significant amount of time to train a large model on a single GPU and there are many alternatives to train large models without sacrificing too much on computation, it does not seem like the paper provides strong enough evidence over choosing an alternative approach. The paper claims that the "feasibility" of training large models as the main motivation, but the memory itself is not the only bottleneck and the paper completely ignores the training speed side of the story.