Compressive sensing with un-trained neural networks: Gradient descent finds the smoothest approximation

Untrained convolutional neural networks have emerged as highly successful tools for image recovery and restoration, for a variety of problems including denoising, compressive sensing, and inpainting [Uly 18; Jin 19; Vee 18; JH19; Hec19; HH19; Bos 20; Wan 20; HA20; Aro 20]. As opposed to trained convolutional neural networks, that learn an image prior from training data, untrained convolutional networks act as an image prior without any training and solely based on the architecture of the network and the optimization procedure used to fit them. The benefit of untrained networks was first observed in the Deep Image Prior (DIP) paper [Uly 18]. The key observation of Ulyanov et al. [Uly 18] is that fitting a standard overparameterized convolutional autoencoder (specifically, the U-net [Ron 15] or variations thereoff) to a single noisy/corrupted image, when combined with early stopping, yields excellent denoising, inpainting, and super-resolution performance. Subsequent literature has demonstrated that many elements of the architecture of a convolutional autoencoder--such as the encoder part--are irrelevant for this behavior to emerge.