Normalization methods play an important role in enhancing the performance of deep learning while their theoretical understandings have been limited.

In the realm of deep learning, the Fisher information matrix (FIM) gives novel insights and useful tools to characterize the loss landscape, perform second-order optimization, and build geometric learning theories. The exact FIM is either unavailable in closed form or too expensive to compute. In practice, it is almost always estimated based on empirical samples. We investigate two such estimators based on two equivalent representations of the FIM -- both unbiased and consistent. Their estimation quality is naturally gauged by their variance given in closed form. We analyze how the parametric structure of a deep neural network can affect the variance. The meaning of this variance measure and its upper bounds are then discussed in the context of deep learning.

Natural gradients have long been studied in deep reinforcement learning due to their fast convergence properties and covariant weight updates. However, computing natural gradients requires inversion of the Fisher Information Matrix (FIM) at each iteration, which is computationally prohibitive in nature. In this paper, we present an efficient and scalable natural policy optimization technique that leverages a rank-1 approximation to full inverse-FIM. We theoretically show that under certain conditions, a rank-1 approximation to inverse-FIM converges faster than policy gradients and, under some conditions, enjoys the same sample complexity as stochastic policy gradient methods. We benchmark our method on a diverse set of environments and show that it achieves superior performance to standard actor-critic and trust-region baselines.

In the realm of deep learning, the Fisher information matrix (FIM) gives novel insights and useful tools to characterize the loss landscape, perform second-order optimization, and build geometric learning theories. The exact FIM is either unavailable in closed form or too expensive to compute. In practice, it is almost always estimated based on empirical samples. We investigate two such estimators based on two equivalent representations of the FIM --- both unbiased and consistent. Their estimation quality is naturally gauged by their variance given in closed form. We analyze how the parametric structure of a deep neural network can affect the variance. The meaning of this variance measure and its upper bounds are then discussed in the context of deep learning.

Most language models (LMs) are trained and applied in an autoregressive left-to-right fashion, predicting the next token from the preceding ones. However, this ignores that the full sequence is available during training.

The dynamics of tumor-immune interactions within a complex tumor microenvironment are typically modeled using a system of ordinary differential equations or partial differential equations. These models introduce some unknown parameters that need to be estimated accurately and efficiently from the limited and noisy experimental data. Moreover, due to the intricate biological complexity and limitations in experimental measurements, tumor-immune dynamics are not fully understood, and therefore, only partial knowledge of the underlying physics may be available, resulting in unknown or missing terms within the system of equations. In this study, we develop a cancer biology-informed neural network model(CBINN) to infer the unknown parameters in the system of equations as well as to discover the missing physics from sparse and noisy measurements. We test the performance of the CBINN model on three distinct nonlinear compartmental tumor-immune models and evaluate its robustness across multiple synthetic noise levels. By harnessing these highly nonlinear dynamics, our CBINN framework effectively estimates the unknown model parameters and uncovers the underlying physical laws or mathematical structures that govern these biological systems, even from scattered and noisy measurements. The models chosen here represent the dynamic patterns commonly observed in compartmental models of tumor-immune interactions, thereby validating the generalizability and efficacy of our methodology.