Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search

This report investigates the problem of learning rate optimisation, focusing on techniques that remove the programmer's burden to choose a proper initial learning rate. The report aims to satisfy two purposes: 1. Acting as an intuition-led guide to Defazio and Mishchenko's 2023 Learning-Rate-Free Learning by D-Adaptation [2] and Mahsereci and Hennig's 2015 Probabilistic Line Searches for Stochastic Optimisation [5]. 2. Presenting a unified notation to discuss optimisation techniques, allowing us to bring together the two learning-rate-free approaches and introduce probabilistics to D-Adaptation in the Discussion section (4). We will begin by recapping the general problem of optimisation. This will establish a common language through which to discuss optimisation algorithms, and introduce the notation used in Defazio et al's D-Adaptation paper.