Online Deep Learning (ODL) and Hedge Back-propagation

As the main concept of deep neural networks is to train through back-propagation in a batch setting, the data is required to be available in an offline setting. As a consequence, the scheme is irrelevant for many practical situations, in which the data arrives in sequence and cannot be stored. ODL is very challenging as it cannot use back-propagation. Two years ago, Sahoo et al (2018) addressed the gap between online learning and deep learning, where they claimed that "without the power of depth, it would be difficult to learn complex patterns". They presented a novel framework for ODL (to be reviewed later).