Self-Supervised Fine-tuning for Image Enhancement of Super-Resolution Deep Neural Networks

--While Deep Neural Networks (DNNs) trained for image and video super-resolution regularly achieve new state-of-the-art performance, they also suffer from significant drawbacks. One of their limitations is their tendency to generate strong artifacts in their solution. This may occur when the low-resolution image formation model does not match that seen during training. Artifacts also regularly arise when training Generative Adversarial Networks for inverse imaging problems. In this paper, we propose an efficient, fully self-supervised approach to remove the observed artifacts. More specifically, at test time, given an image and its known image formation model, we fine-tune the parameters of the trained network and iteratively update them using a data consistency loss. We apply our method to image and video super-resolution neural networks and show that our proposed framework consistently enhances the solution originally provided by the neural network. In the past decade, the application of Deep Neural Networks (DNNs) to solving inverse imaging problems has gained considerable popularity [ 2 ]. The observed image y is assumed to come from a known image formation model with degradation operator A, which we formulate here as y Ax ǫ, where ǫ denotes the noise. The parameters ψ are learned through a lengthy training stage which requires the use of a large dataset of input-output (y, x) pairs. The training data is commonly generated by applying the degradation operator A to the clean images to obtain the corresponding degraded images used for training. With this straightforward framework combined with the fast-growing nature of Deep Learning, new state-of-the-art results for image restoration tasks are regularly achieved. Preliminary results of this work were presented at the 2019 IEEE International Conference on Image Processing (ICIP) [ 1 ].