TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning

Existing work solves this problem by reducing the number of trainable parameters. However, this doesn't directly translate to memory saving since the major bottleneck is the activations, not parameters.