An Application of the Holonomic Gradient Method to the Neural Tangent Kernel

Each of these expectations is called a dual activation of σ and its derivative σ respectively. Note that these expectations can be expressed as definite integrals with parameters. Attempts have been made to calculate these expectations for various activator functions, and closed forms have been found for many activator functions. Han et al [8] gives several new closed forms as well as a survey on the works on closed forms. A system of linear partial differential equations of n variables is called a holonomic system when the dimension of its characteristic variety (the variety defined by the ideal generated by principal symbols) is n. A distribution is called a holonomic distribution if it is a solution of a holonomic system. In this paper, we note that when the activator function is a holonomic distribution, these expectations satisfy holonomic systems of linear partial differential equations and further show that these holonomic systems can be derived automatically by computer algebraic algorithms. We give the following new results based on this fact.