Online Learning of Neural Networks

We study online learning of feedforward neural networks with the sign activation function that implement functions from the unit ball in Rd to a finite label set Y = {1,...,Y}. First, we characterize a margin condition that is sufficient and in some cases necessary for online learnability of a neural network: Every neuron in the first hidden layer classifies all instances with some margin γ bounded away from zero. Quantitatively, we prove that for any net, the optimal mistake bound is at most approximately TS(d,γ), which is the (d,γ)-totally-separablepacking number, a more restricted variation of the standard (d,γ)-packing number. We complement this result by constructing a net on which any learner makes TS(d,γ) many mistakes. We also give a quantitative lower bound of approximately TS(d,γ) max{1/(γ d)d,d} when γ 1/2, implying that for some nets and input sequences every learner will err for exp(d) many times, and that a dimension-free mistake bound is almost always impossible.