The background model is trained byperturbing aproportion ofrandomly chosen pixels,wheretheperturbation isdonebyreplacing the pixel value by a uniformly sampled random value between 0 and 255.

Deep probabilistic generative models enable modeling the likelihoods of very high dimensional data. An important application of generative modeling should be the ability to detect out-of-distribution (OOD) samples by setting a threshold on the likelihood. However, a recent study shows that probabilistic generative models can, in some cases, assign higher likelihoods on certain types of OOD samples, making the OOD detection rules based on likelihood threshold problematic. To address this issue, several OOD detection methods have been proposed for deep generative models. In this paper, we make the observation that some of these methods fail when applied to generative models based on Variational Auto-encoders (VAE). As an alternative, we propose Likelihood Regret, an efficient OOD score for VAEs. We benchmark our proposed method over existing approaches, and empirical results suggest that our method obtains the best overall OOD detection performances compared with other OOD method applied on VAE.

We thank all reviewers for their useful comments. Table 1: AUCROC obtained from Likelihood Regret on Glow and PixelCNN. We now address detailed concerns of each reviewer. We apologize for the confusion caused by the notation. Please refer to items 1-3 for concerns regarding why we focus on V AE's OOD detection.

Weaknesses: My main concern is on the lack of comparison with previous methods (input complexity adjusted score and the likelihood ratio method) on the corresponding models(e.g. Those methods were developed and tested with the corresponding generative models in the original paper and it seems unfair to only compare with their method on VAEs. Without this comparison, if a researcher wants to choose the SOTA OOD detection method for their own applications, it's hard to tell which method will most likely achieve the best performance if they have the freedom to choose their own generative models. This is the main drawback and the main reason for my rating. Furthermore, this leads to the general motivation of the paper. I really like the analysis on why prior likelihood-ratio based methods didn't work as well on VAEs, however, if all we care about is detecting OOD examples, why is it absolutely necessary to have a method that works well on VAEs?

The key idea of this paper is checking how well a given VAE can be further trained on a given test input. The hope is that training the encoder for further iterations may increase the likelihood for OOD samples more when compared to inliers to facilitate detection. The authors characterize this improvement by a measure, coined as likelihood regret. The authors do not provide any analysis why this method might work, or characterize the conditions when it might not. This is not a requirement, but then the paper should provide enough empirical evidence that the approach is noteworthy.

Deep probabilistic generative models enable modeling the likelihoods of very high dimensional data. An important application of generative modeling should be the ability to detect out-of-distribution (OOD) samples by setting a threshold on the likelihood. However, a recent study shows that probabilistic generative models can, in some cases, assign higher likelihoods on certain types of OOD samples, making the OOD detection rules based on likelihood threshold problematic. To address this issue, several OOD detection methods have been proposed for deep generative models. In this paper, we make the observation that some of these methods fail when applied to generative models based on Variational Auto-encoders (VAE). As an alternative, we propose Likelihood Regret, an efficient OOD score for VAEs.

Deep probabilistic generative models enable modeling the likelihoods of very high dimensional data. An important application of generative modeling should be the ability to detect out-of-distribution (OOD) samples by setting a threshold on the likelihood. However, a recent study shows that probabilistic generative models can, in some cases, assign higher likelihoods on certain types of OOD samples, making the OOD detection rules based on likelihood threshold problematic. To address this issue, several OOD detection methods have been proposed for deep generative models. In this paper, we make the observation that some of these methods fail when applied to generative models based on Variational Auto-encoders (VAE). As an alternative, we propose Likelihood Regret, an efficient OOD score for VAEs. We benchmark our proposed method over existing approaches, and empirical results suggest that our method obtains the best overall OOD detection performances compared with other OOD method applied on VAE.