Consider the following system with a generic R.H.S-X We also added the approach of Blackbox [25], which also deals with a combinatorial optimization problem with a linear objective.

Global pooling is one of the most significant operations in many machine learning models and tasks, whose implementation, however, is often empirical in practice. In this study, we develop a novel and solid global pooling framework through the lens of optimal transport. We demonstrate that most existing global pooling methods are equivalent to solving some specializations of an unbalanced optimal transport (UOT) problem. Making the parameters of the UOT problem learnable, we unify various global pooling methods in the same framework, and accordingly, propose a generalized global pooling layer called UOT-Pooling (UOTP) for neural networks. Besides implementing the UOTP layer based on the classic Sinkhorn-scaling algorithm, we design a new model architecture based on the Bregman ADMM algorithm, which has better numerical stability and can reproduce existing pooling layers more effectively. We test our UOTP layers in several application scenarios, including multi-instance learning, graph classification, and image classification. Our UOTP layers can either imitate conventional global pooling layers or learn some new pooling mechanisms leading to better performance.

While several methods have been proposed to tackle this problem, there is limited work explaining why these methods work well. This paper has the goal of better explaining a popularly used technique for avoiding catastrophic forgetting: quadratic regularization. We show that quadratic regularizers prevent forgetting of past tasks by interpolating current and previous values of model parameters at every training iteration. Over multiple training iterations, this interpolation operation reduces the learning rates of more important model parameters, thereby minimizing their movement. Our analysis also reveals two drawbacks of quadratic regularization: (a) dependence of parameter interpolation on training hyperparameters, which often leads to training instability and (b) assignment of lower importance to deeper layers, which are generally the place forgetting occurs in DNNs. Via a simple modification to the order of operations, we show these drawbacks can be easily avoided, resulting in 6.2% higher average accuracy at 4.5% lower average forgetting. We confirm the robustness of our results by training over 2000 models in different settings. Learning algorithms are often designed under the assumption of independent and identical data distributions. However, this assumption is violated in several practical scenarios, such as continual learning and lifelong learning, where data distributions evolve constantly. In such settings, deep neural networks (DNNs) witness catastrophic forgetting and have difficulty adapting to new tasks without losing performance on previously learned ones. Several past works have tried to address this problem. While these works have shown promising results, a detailed understanding of their proposed methods is still to be developed. Understanding the reasons due to which existing methods for preventing catastrophic forgetting work or fail can open the possibility of developing better methods. With this motivation, in this work, we analyze quadratic regularization, a popular technique for preventing catastrophic forgetting in DNNs.