Online Learning from Finite Training Sets: An Analytical Case Study