A Dual Optimization View to Empirical Risk Minimization with f-Divergence Regularization

--The dual formulation of empirical risk minimization with f -divergence regularization (ERM-f DR) is introduced. The solution of the dual optimization problem to the ERM-f DR is connected to the notion of normalization function introduced as an implicit function. This dual approach leverages the Legendre-Fenchel transform and the implicit function theorem to provide a nonlinear ODE expression to the normalization function. Furthermore, the nonlinear ODE expression and its properties provide a computationally efficient method to calculate the normalization function of the ERM-f DR solution under a mild condition. Empirical risk minimization (ERM) [1]-[6] is often posed as an optimization problem regularized by a statistical distance between the probability measure to be optimized and a given reference measure [7]-[13].