Asia
Loss Decoupling for Task-Agnostic Continual Learning Y an-Shuo Liang and Wu-Jun Li
Continual learning requires the model to learn multiple tasks in a sequential order.
2 Frameworkandassumptions 2.1 Stochasticoptimizationundertimedrift ThroughoutSections2-4,weconsiderthesequenceofstochasticoptimizationproblems min
Our results concisely explain the interplay between the learning rate, the noise variance in the gradient oracle, and the strength ofthetime drift. The high-probability results merely assume that thegradient noise and time drift have light tails. Moreover, none of the results require the objectives to have bounded domains.
2 Frameworkandassumptions 2.1 Stochasticoptimizationundertimedrift Weconsiderthesequenceofstochasticoptimizationproblems min
Our results concisely explain the interplay between the learning rate, the noise variance in the gradient oracle, and the strength ofthetime drift. The high-probability results merely assume that thegradient noise and time drift have light tails. Moreover, none of the results require the objectives to have bounded domains.