Features extracted from deep layers of classification networks are widely used as image descriptors. Here, we exploit an unexplored property of these features: their internal dissimilarity. While small image patches are known to have similar statistics across image scales, it turns out that the internal distribution of deep features varies distinctively between scales. We show how this deep self dissimilarity (DSD) property can be used as a powerful visual fingerprint. Particularly, we illustrate that full-reference and no-reference image quality measures derived from DSD are highly correlated with human preference. In addition, incorporating DSD as a loss function in training of image restoration networks, leads to results that are at least as photo-realistic as those obtained by GAN based methods, while not requiring adversarial training.

We consider the problem of learning mixtures of Gaussians under the constraint of approximate differential privacy. We prove that eO(k2dlog3/2(1/δ)/α2ε) samples are sufficient to learn a mixture of k axis-aligned Gaussians in Rd to within total variation distance αwhile satisfying (ε,δ)-differential privacy. This is the first result for privately learning mixtures of unbounded axis-aligned (or even unbounded univariate) Gaussians. If the covariance matrices of each of the Gaussians is the identity matrix, we show that eO(kd/α2 + kdlog(1/δ)/αε) samples are sufficient. To prove our results, we design a new technique for privately learning mixture distributions. A class of distributions F is said to be list-decodable if there is an algorithm that, given "heavily corrupted" samples from f F, outputs a list of distributions one of which approximates f. We show that if F is privately list-decodable then we can learn mixtures of distributions in F. Finally, we show axis-aligned Gaussian distributions are privately list-decodable, thereby proving mixtures of such distributions are privately learnable.

In what follows, we give some details of content omitted in the paper due to space limit. The supplements are organized as follows. We give some proof of Lemma 1, 2, Proposition 1, Lemma 3, 4, and Theorem 2 in Section A.1 -A.6, respectively. We provide some training details in Section A.13 as well as experiment details and results in Section A.14. We compare polynomial CBFs with NCBF in A.15, compare NCBFs with different activation functions in A.16. A.1 Proof of Lemma 1 We prove by induction on L. If L =1, then x 2 X(S) if the pre-activation input to the (1,j) neuron is nonnegative for all j 2 S1 and nonpositive for all j/2 S1. We have that the pre-activation input is equal to WT1jx+r1j, establishing the result for L =1 .

By proof of Theorem 1, we have HCIL(x) = HWP(x)+HTP(x). Equal contribution The work was done when this author was visiting Bing Liu's group at University of Illinois at Chicago Correspondance author. According to proof of Theorem 4 ii), we have HTP(x) η. Note that ODIN is not applicable to iCaRL and Mnemonics as they are not based on softmax but some distance functions. The result for C100-10T are reported in the main paper. For the postprocessing method ODIN, we only reported the results on C100-10T due to space limitations. Tab. 5 shows the results on the other datasets. A continual learning method with a better AUC shows a better CIL performance than other methods with lower AUC.