Explained: Deep Learning in Tensorflow -- Chapter 0

There are several ways of finding optimal LR through scheduling: Power, Icycle and exponential scheduling. One of the ways is to train the model for a few hundred iterations, exponentially increasing the LR from very small value to a very large value and hen looking at the learning curve picking a learning rate slightly lower than the one at which the learning curve starts shooting back up. As in the LR vs Loss curve on the left side, around 1/10 is the optimal LR. "Essentially, all models are wrong, but some are useful." When more than one neurons are stacked over each other where connections between the units do not form a cycle, then it forms a single layer of dense neural network. When one or more than one layer is arranged parallel to each other (may have a different number of units of neurons) in which one being the input and output layer at the beginning and ending respectively, and another being the hidden layer in between these two layers such that information (output from each neuron) moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes, it is known as feed-forward neural network.