Some observations concerning Off Training Set (OTS) error

A new measure of generalisation error called Off Training Set (OTS) er ror was introduced recently in [Wolpert, 1996b, Wolpert, 1996a]. Under quit e weak assumptions it was shown that with respect to OTS error there are no a priori distinctions between learning algorithms, at least if it is assumed that the target functions are uniformly distributed. Thus, as far as OTS error is co ncerned, an algorithm that minimizes error on the training set will do no better tha n random guessing. If OTS error accurately models the concept of generaliz ation then this is a depressing conclusion indeed. However, in this paper it is argued that OTS error does not model wh at is normally meant by generalization error. In particular, it is shown th at the assumptions underlying one of the main "no free lunch" (NFL) theor ems (theorem 2) in [Wolpert, 1996b] imply that the distributions used to genera te training data and testing data have disjoint supports. Thus, training a neu ral network to recognise faces by showing it images of handwrittten character s is the kind of learning problem covered by the NFL theorem.