Are line search methods used in deep learning? Why not?

The tutorials talk about gradient descent presumably because it is one of the simplest algorithms used for optimization, so it is easy to explain. Since most of such tutorials are rather brief, they focus on simple stuff. There are at least several popular optimization algorithms beyond simple gradient descent that are used for deep learning. Actually people often use different algorithms then gradient descent since they usually converge faster. Some of them have non-constant learning rate (e.g.