bluesky314/Cyclical_LR_Scheduler_With_Decay_Pytorch

Reach multiple minimas to create a powerful ensemble or just to find the best one using Cyclical Learning Rates with Decay. Ideally decay milestones should intersect with cyclical milestones for smooth transition as shown below. Can be used with any optimizer such as Adam. Sample - (follow similarly for CyclicLinearLR) milestones specifies when learning rate should shoot back up and decay_milestones when learning rate should be decayed. It can be called even in every batch.