Surrogate modeling plays a crucial role in simplifying and approximating complex physical models, making them suitable for large-scale simulations and optimization studies of industrial relevance. Machine learning models, such as neural networks (NNs), are particularly well-suited for this purpose due to their simplicity and strong regression capabilities [1]. However, despite exceptional advancements in machine learning, issues and skepticism regarding the black-box nature and physical inconsistency of these models hinder the adoption of machine learning-based tools (and, more broadly, artificial intelligence) in industrial applications [2, 3]. To mitigate this limitation, significant research has been carried out to enforce known mechanistic relationships between inputs and predictions in NNs. Soft-constrained neural networks represent an approach in which physical equations are included as penalty terms in the loss function [4, 5].

We consider a class of learning problem of point estimation for modeling high-dimensional nonlinear functions, whose learning dynamics is guided by model training dataset, while the estimated parameter in due course provides an acceptable prediction accuracy on a different model validation dataset. Here, we establish an evidential connection between such a learning problem and a hierarchical optimal control problem that provides a framework how to account appropriately for both generalization and regularization at the optimization stage. In particular, we consider the following two objectives: (i) The first one is a controllability-type problem, i.e., generalization, which consists of guaranteeing the estimated parameter to reach a certain target set at some fixed final time, where such a target set is associated with model validation dataset. (ii) The second one is a regularization-type problem ensuring the estimated parameter trajectory to satisfy some regularization property over a certain finite time interval. First, we partition the control into two control strategies that are compatible with two abstract agents, namely, a leader, which is responsible for the controllability-type problem and that of a follower, which is associated with the regularization-type problem. Using the notion of Stackelberg's optimization, we provide conditions on the existence of admissible optimal controls for such a hierarchical optimal control problem under which the follower is required to respond optimally to the strategy of the leader, so as to achieve the overall objectives that ultimately leading to an optimal parameter estimate. Moreover, we provide a nested algorithm, arranged in a hierarchical structure-based on successive approximation methods, for solving the corresponding optimal control problem. Finally, we present some numerical results for a typical nonlinear regression problem.

Zeitschrift fur Operations Research, Serie A, 20 (5), 203-208.