Optimization in Deep Learning

Summary Full-batch GD SGD Momentum SGD NAG AdaGrad RMSProp Adam Speed up by momentum Adaptive learning rate 30. More deep learning coming up! • Optimization in Deep learning (today's session) • Behind AlphaGo • Mastering the game of Go with deep neural networks and tree search • Attention network • Application of Deep Learning and showcase • Any proposal?