We generalize the universal approximation theorem for functional input neural networks (FNN) to differentiable maps by including the approximation of the derivatives. A FNN maps the input from a possibly infinite-dimensional weighted manifold to the real-valued hidden layer, on which a non-linear scalar activation function is applied, and then returns the output into a Banach space via some linear readouts. By proving a weighted Nachbin theorem, we establish a universal approximation theorem for differentiable maps, which goes beyond the usual formulation on compact sets and also includes the approximation of the derivatives. This leads us to approximation results for non-anticipative functionals including the horizontal and vertical derivatives. As a further application, we show that linear functions of the signature are able to approximate path space functionals including their directional derivatives.

Awordvectorspace-sometimes referred toasawordembedding-associates similar wordsina vocabularywithsimilar vectors.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

This appendix is organized as follows: in Section B, C and D we provide the missing proofs of Theorems 3, 4 and 5. In Section E we provide detailed version of Theorems 6 and 7 containing all constants. In Section F we provide a version of Theorem 2 with all constants for completeness. In this section we provide the missing proof of Theorem 3, restated below: Lemma 3. Let W be a real vector space and So now it remains only to show the regret bound. Now we recall the following consequence of concavity of the square root function (see Auer et al. [2002], Duchi et al. [2010] for proofs): for any sequence non-negative numbers x In this section we provide the missing proof of Theorem 4, restated below: Lemma 4. Now it remains to use the regret bound on A. Observe that |s In this section, we provide the missing proof of Theorem 5, restated below: Theorem 5.

Let Z be the set of all indices` [n] such thatα[`] 6= 0. Let Z be the set of all indices` Z such that the`th variable is constrained to be integral.