Weaknesses: * Missing comparison against persistent sampling: This paper proposes to use short-run MCMC to sample from both the prior and true posterior. In practice, since we have only one prior distribution, sampling from the prior can be also done using persistent sampling which often improves the performance of EBMs by a large margin. It's not clear why the proposed method uses short-run MCMC that can potentially mix slowly and can introduce sampling error. Moreover, Eq. 13 shows that sampling error turns the objective into an upper bound on the log-likelihood. This can be dangerous as the model may start increasing the gap between the distribution of approximate samples and the EBM prior by making the distribution harder to sample from.

Weaknesses: While the problem setting and proposed approach are interesting there are some drawbacks in the execution of this idea. First much of the experimental detail is left to the supplementary material and makes the main paper appear lacking in results. Concerningly, few of the transductive baselines outperform the main baselines (see Cora table in the Appendix for Deepwalk features) reported in the main body of the paper and thus their omission is questionable. Furthermore, the chosen datasets as the paper recognizes are either small graphs or contain only a single graph and as a result its difficult to assess how scalable the proposed approach is to larger real world graphs. The biggest weakness in this reviewers opinion is that its unclear why the MCMC scheme proposed is a natural or superior choice to existing approaches to training EBMs in the literature. Training EBMs have seen a resurgence of late and there have been multiple approaches that provide significant computational benefit [1] [2] [3] are few recent examples.

Energy-Based Models (EBMs) have been widely used for generative modeling. Contrastive Divergence (CD), a prevailing training objective for EBMs, requires sampling from the EBM with Markov Chain Monte Carlo methods (MCMCs), which leads to an irreconcilable trade-off between the computational burden and the validity of the CD. Running MCMCs till convergence is computationally intensive. On the other hand, short-run MCMC brings in an extra non-negligible parameter gradient term that is difficult to handle. In this paper, we provide a general interpretation of CD, viewing it as a special instance of our proposed Diffusion Contrastive Divergence (DCD) family. By replacing the Langevin dynamic used in CD with other EBM-parameter-free diffusion processes, we propose a more efficient divergence. We show that the proposed DCDs are both more computationally efficient than the CD and are not limited to a non-negligible gradient term. We conduct intensive experiments, including both synthesis data modeling and high-dimensional image denoising and generation, to show the advantages of the proposed DCDs. On the synthetic data learning and image denoising experiments, our proposed DCD outperforms CD by a large margin. In image generation experiments, the proposed DCD is capable of training an energy-based model for generating the Celab-A $32\times 32$ dataset, which is comparable to existing EBMs.

Deep energy-based models (EBMs) are very flexible in distribution parametrization but computationally challenging because of the intractable partition function. They are typically trained via maximum likelihood, using contrastive divergence to approximate the gradient of the KL divergence between data and model distribution. While KL divergence has many desirable properties, other f-divergences have shown advantages in training implicit density generative models such as generative adversarial networks. In this paper, we propose a general variational framework termed f-EBM to train EBMs using any desired f-divergence. We introduce a corresponding optimization algorithm and prove its local convergence property with non-linear dynamical systems theory. Experimental results demonstrate the superiority of f-EBM over contrastive divergence, as well as the benefits of training EBMs using f-divergences other than KL.