L$^2$C -- Learning to Learn to Compress