DIFAR: Deep Image Formation and Retouching

Given (a) poorly exposed image, DIF AR(c) produces an image with pleasing contrast and colour better matching the groundtruth (d) compared to the state-of-the-art DeepUPE model [42] (b). Abstract W e present a novel neural network architecture for the image signal processing (ISP) pipeline. In a camera system, the ISP is a critical component that forms a high quality RGB image from RA W camera sensor data. Typical ISP pipelines sequentially apply a complex set of traditional image processing modules, such as demosaicing, denoising, tone mapping, etc. W e introduce a new deep network that replaces all these modules, dubbed Deep Image Formation And Retouching (DIFAR) . DIF AR introduces a multi-scale context-aware pixel-level block for local de-noising/demosaicing operations and a retouching block for global refinement of image colour, luminance and saturation. DIF AR can also be trained for RGB to RGB image enhancement. DIF AR is parameter-efficient and outperforms recently proposed deep learning approaches in both objective and perceptual metrics, setting new state-of-the-art performance on multiple datasets including Samsung S7 [38] and MIT-Adobe 5k [6]. 1. Introduction Image quality is of fundamental importance in any imaging system, including DSLR and smartphone cameras. At the imaging sensor, RA W data is normally captured on a color filter array (such as the well-known Bayer pattern) where at each pixel, only a red, green, or blue color is available. This mosaiced RA W data suffers from noise, vignetting, lack of white balance, and many other defects and additionally has a high dynamic range. The camera's image signal processing (ISP) pipeline is responsible for forming a high quality RGB image with minimal noise, pleasing colors, sharp detail, and good contrast from the degraded RA W data. In most cases, the ISP is realised as a modular sequence of traditional image signal processing algorithms (Figure 2) each responsible for a single well-defined image operation (e.g.