Fast.ai: What I Learned from Lessons 1–3 – Hacker Noon

Lessons 2 and 3 link to other content to read. Stanford's CS231n Convolutional Neural Networks for Visual Recognition Neural Networks Part 1 lecture has a lot of interesting ideas. Chapter 3 of Neural Networks and Deep Learning textbook has interesting insights into the history of the development of cost functions. If you use a mean squared error cost function with a sigmoid activation function in the output layer then the gradient of the cost function w.r.t. This derivative gets very low values in case the input of the sigmoid function is far from 0. It leads to slower learning because gradient descent will make small updates to weights in this case.