Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm

Abstract--Targeting solutions over'flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged Various approaches have also been proposed to further I. Unfortunately, Advances in deep neural network (DNN) architectures have a unifying framework is lacking to encompass existing SAM led to impressive success across various domains including variants, and inspire the principled design of novel approaches. Owing to the markedly Toward this goal, the present work relies on preconditioning high dimensionality, DNNs can memorize a large gamut of to unify SAM variants; hence, the term preconditioned training data [4]. As a result, small loss during training does (pre) SAM. Depending on where preconditioning is not guarantee generalization to unseen data.