Several machine learning applications involve the optimization of higher-order derivatives (e.g., gradients of gradients) during training, which can be expensive with respect to memory and computation even with automatic differentiation. As a typical example in generative modeling, score matching~(SM) involves the optimization of the trace of a Hessian. To improve computing efficiency, we rewrite the SM objective and its variants in terms of directional derivatives, and present a generic strategy to efficiently approximate any-order directional derivative with finite difference~(FD). Our approximation only involves function evaluations, which can be executed in parallel, and no gradient computations. Thus, it reduces the total computational cost while also improving numerical stability. We provide two instantiations by reformulating variants of SM objectives into the FD forms. Empirically, we demonstrate that our methods produce results comparable to the gradient-based counterparts while being much more computationally efficient.

Additional Feedback: Concerning the title (and I know this is maybe pedantic), 'Efficient' is an entirely relative term, and means little in the absolute context of a paper title. An alternative title'Generative Modeling via Finite-Difference Score Matching' is both clearer and more concise. What does it mean to perform learning in a generative modeling'efficiently'? The proposed methods achieve efficiency with respect to the previous implementations of SSM and DSM, and are strictly in that sense efficient. Including'efficient' in the title only serves to obfuscate the actual contribution of the paper, which isn't about efficient learning of generative models compared to all other generative model learning approaches, but efficient score matching by finite difference methods specifically.

The authors reformulate denoising score matching and sliced score matching in terms of directional derivatives (1st and 2nd order respectively) and show that estimating these derivatives using finite differences (FD) leads to faster training and lower memory usage while yielding comparable results to using exact derivatives. The paper also introduces a general approach to estimating directional derivatives of any order using FDs of the function. The reviewers found the paper interesting and very well written. They also appreciated the extensive evaluation of the algorithms on a range of different generative models as well as the clarifications provided in the rebuttal. There were some concerns however about the novelty as well as the necessity of the developed FD estimation approach for directional derivatives.

Several machine learning applications involve the optimization of higher-order derivatives (e.g., gradients of gradients) during training, which can be expensive with respect to memory and computation even with automatic differentiation. As a typical example in generative modeling, score matching (SM) involves the optimization of the trace of a Hessian. To improve computing efficiency, we rewrite the SM objective and its variants in terms of directional derivatives, and present a generic strategy to efficiently approximate any-order directional derivative with finite difference (FD). Our approximation only involves function evaluations, which can be executed in parallel, and no gradient computations. Thus, it reduces the total computational cost while also improving numerical stability.