InterpretableLightweightTransformerviaUnrolling ofLearnedGraphSmoothnessPriors

Orthogonally, algorithm unrolling[14] implements iterations of a model-based algorithm as a sequence of neural layers to build afeed-forward network, whose parameters can be learned endto-end via back-propagation from data. A classic example is the unrolling of theiterative soft-1While works existtoanalyze existing transformer architectures [5,6,7,8,9],only [10,11]characterized the performance ofasingle self-attention layer and ashallowtransformer,respectively.