Reviews: Overcoming Catastrophic Forgetting by Incremental Moment Matching

Not including an objective evaluation of limitations is a flaw of this otherwise well written paper, especially when the method relies crucially on weight transfer (as the authors point out outside the main paper, i.e. supplementary text and rebuttal). However, weight transfer is known to be an inadequate initialization technique between different problem classes and the authors don't clearly address this issue, nor do they properly qualify the applicability of the method. In balance, this paper does give sufficient evidence that weight transfer and some form of parameter averaging are promising directions of future investigation, at least in a subset of interesting cases. The method is thoroughly benchmarked, in several incarnations, against state-of-the-art baselines on standard'toy' problems defined on top of MNIST, as well as more challenging ImagNet2CUB and the Lifelog dataset. A new parameterization, dubbed'drop-transfer' is proposed as an alternative to standard weight initialization of model parameters on new tasks.