Inspired by BatchNorm, there has been an explosion of normalization layers in deep learning.

However, the quantization functions used in most conventional quantization methods are non-differentiable, which increases the optimization difficulty ofquantized networks. Compared with full-precision parameters (i.e.,32-bit floating numbers), low-bit values areselected from amuch smaller set. For example, there are only 16 possibilities in 4-bit space. Thus, we present to regard the discrete weights in an arbitrary quantized neural network as searchable variables, and utilize a differential method to search them accurately. In particular, each weight is represented as a probability distribution over the discrete value set. The probabilities are optimized during training and the values with the highest probability are selected toestablish the desired quantizednetwork.

Solving inverse problems, such as parameter estimation and optimal control, is a vital part of science. Many experiments repeatedly collect data and rely on machine learning algorithms to quickly infer solutions to the associated inverse problems. Wefind that state-of-the-art training techniques are not well-suited to many problems that involve physical processes. The highly nonlinear behavior, common in physical processes, results in strongly varying gradients that lead first-order optimizers like SGD or Adam to compute suboptimal optimization directions. We propose a novel hybrid training approach that combines higherorder optimization methods with machine learning techniques.

Machine learning models will often fail when deployed in an environment with a data distribution that is different than the training distribution.