From Posterior Sampling to Meaningful Diversity in Image Restoration

Image restoration problems are typically ill-posed in the sense that each degraded image can be restored in infinitely many valid ways. To accommodate this, many works generate a diverse set of outputs by attempting to randomly sample from the posterior distribution of natural images given the degraded input. Here we argue that this strategy is commonly of limited practical value because of the heavy tail of the posterior distribution. Consider for example inpainting a missing region of the sky in an image. Since there is a high probability that the missing region contains no object but clouds, any set of samples from the posterior would be entirely dominated by (practically identical) completions of sky. In this paper, we initiate the study of meaningfully diverse image restoration. We explore several post-processing approaches that can be combined with any diverse image restoration method to yield semantically meaningful diversity. Moreover, we propose a practical approach for allowing diffusion based image restoration methods to generate meaningfully diverse outputs, while incurring only negligent computational overhead. We conduct extensive user studies to analyze the proposed techniques, and find the strategy of reducing similarity between outputs to be significantly favorable over posterior sampling. Image restoration is a collective name for tasks in which a corrupted or low resolution image is restored into a better quality one. Example tasks include image inpainting, super-resolution, compression artifact reduction and denoising. Common to most image restoration problems is their ill-posed nature, which causes each degraded image to have infinitely many valid restoration solutions. Depending on the severity of the degradation, these solutions may differ significantly, and often correspond to diverse semantic meanings (Bahat & Michaeli, 2020). In the past, image restoration methods were commonly designed to output a single solution for each degraded input (Haris et al., 2018; Kupyn et al., 2019; Liang et al., 2021; Pathak et al., 2016; Wang et al., 2018; Zhang et al., 2017).