Adaptive Learning with Binary Neurons

Lab oratoire TIMC-IMA G (UMR CNRS-UJF 5525) Domaine de La Mer i - Bât. Jean Roget 38706 La T ron he, F ran e F ebruary 19, 2018 Abstra t A e ien t in remen tal learning algorithm for lassi ation tasks, alled NetLines, w ell adapted for b oth binary and real-v alued input patterns is presen ted. It generates small ompa t feedforw ard neural net w orks with one hidden la y er of binary units and binary output units. A on v ergen e theorem ensures that solutions with a nite n um b er of hidden units exist for b oth binary and real-v alued input patterns. An implemen tation for problems with more than t w o lasses, v alid for an y binary lassi er, is prop osed. The generalization error and the size of the resulting net w orks are ompared to the b est published results on w ell-kno wn lassi ation b en hmarks. The relationship b et w een n um b er of w eigh ts, learning apa it y and net w ork's generalization abilit y is w ell understo o d only for the simple p er eptron, a single binary unit whose output is a sigmoidal fun tion of the w eigh ted sum of its inputs. In this 1 ase, e ien t learning algorithms based on theoreti al results allo w the determination of the optimal w eigh ts. Ho w ev er, simple p er eptrons an only generalize those (v ery few) problems in whi h the input patterns are line arly sep ar able (LS).