Competition Among Networks Improves Committee Performance

ABSTRACT The separation of generalization error into two types, bias and variance (Geman, Bienenstock, Doursat, 1992), leads to the notion of error reduction by averaging over a "committee" of classifiers (Perrone, 1993). Committee perfonnance decreases with both the average error of the constituent classifiers and increases with the degree to which the misclassifications are correlated across the committee. Here, a method for reducing correlations is introduced, that uses a winner-take-all procedure similar to competitive learning to drive the individual networks to different minima in weight space with respect to the training set, such that correlations in generalization perfonnance will be reduced, thereby reducing committee error. 1 INTRODUCTION The problem of constructing a predictor can generally be viewed as finding the right combination of bias and variance (Geman, Bienenstock, Doursat, 1992) to reduce the expected error. Since a neural network predictor inherently has an excessive number of parameters, reducing the prediction error is usually done by reducing variance. Methods for reducing neural network complexity can be viewed as a regularization technique to reduce this variance.