Some Best Practices in Operator Learning

A neural operator is a neural network that is intended to approximate an operator between function spaces [1-3]. An example of an output function for a neural operator is a solution to a differential equation. Examples of input functions for a neural operator are the initial conditions or the boundary conditions for the differential equation. The study of neural operators is called operator learning. Various choices of hyperparameters and training strategies are made when training a neural network. Limiting such choices to known robust choices can significantly improve the speed of hyperparameters searches. This paper studies the effect of some of these choices in operator learning, that is, it tries to find some best practices in operator learning. This is not a comparison between different architectures. In order, this paper studies the choice of activation function, the use of dropout, the use of stochastic weight averaging and the use of a learning rate finder.