Efficient Differentiable Programming in a Functional Array-Processing Language

EPFL, Switzerland We present a system for the automatic differentiation of a higher-order functional array-processing language. The core functional language underlying this system simultaneously supports both sourceto-source automatic differentiation and global optimizations such as loop transformations. Thanks to this feature, we demonstrate how for some real-world machine learning and computer vision benchmarks, the system outperforms the state-of-the-art automatic differentiation tools. This investigation led him to see the importance of functional arguments and recursive functions in the field of symbolic computation. From Norvig [38, p248]. 1 INTRODUCTION Functional programming (FP) and automatic differentiation (AD) have been natural partners for sixty years, and major functional languages all have elegant automatic differentiation packages [6, 17, 29]. With the increasing importance of numerical engineering disciplines such as machine learning, speech processing, and computer vision, there has never been a greater need for systems which mitigate the tedious and error-prone process of manual coding of derivatives. However the popular packages (TensorFlow, CNTK) all implement clunky (E)DSLs in procedural languages such as Python and C . One reason is that the FP packages are slower than their imperative counterparts, by many orders of magnitude [48], because modern applications depend heavily on array processing, with vectors, matrices, and tensors as the canonical datatypes. In contrast, AD for FP has generally handled only scalar workloads efficiently [29]. Our key contribution in this paper is to take a recently introduced F# subset designed for efficient compilation of array-processing workloads, and to augment it with vector AD primitives, yielding a functional AD tool that is competitive with the best C/C and Fortran tools on many benchmarks, and considerably faster on others.