Nonlinearity Enhanced Adaptive Activation Function

While neural networks (NN) were first proposed in 1943 [1], initial implementations were restricted to networks with a small number of neurons and one or two layers[2], [3]. This limitation was eliminated through the backpropagation training algorithm[3], [4], [5] in conjunction with exponential improvements in computational performance. The resulting procedure generates a system model exclusively from experimental or simulated data and can accordingly be employed in a wide variety of scientific and engineering fields. In particular, a system, which typically can be characterized by a few coordinates and equations, is instead described by a large number of variables that interact nonlinearly. By optimizing a loss function, which may be further subject to physical constraints as in physics-informed machine 1 learning,[6] the parameters associated with the interactions are adjusted to approximate the data. The trained model then can predict the response of the system to unobserved input data. Although such an approach possesses significant advantages in terms of generality and simplicity, it lacks the precision and efficiency afforded by the solution of deterministic equations. Similarly, the large dimensionality of the representation obscures the underlying physics and mathematics. For complex systems, however, especially in the presence of stochastic noise or measurement inaccuracy, procedures based on numerical optimization can be effectively optimal.[7],