SIMPLIFYING NEURAL NETS BY DISCOVERING FLAT MINIMA

We present a new algorithm for finding low complexity networks with high generalization capability. The algorithm searches for large connected regions of so-called ''fiat'' minima of the error function. Inthe weight-space environment of a "flat" minimum, the error remains approximately constant. Using an MDL-based argument, flatminima can be shown to correspond to low expected overfitting. Although our algorithm requires the computation of second order derivatives, it has backprop's order of complexity.