Reviews: Regularization Learning Networks: Deep Learning for Tabular Datasets

Summary This work develops Regularization Learning Networks (RLN), an approach to learn varying levels of regularization for different network weights. Regularization levels for (D)NN are usually a global hyper-parameter, optimized via CV for the whole network. The author point out that figuring different levels or regularization for different features is appealing especially for data where features are of different "type" with some possibly highly important while others not and the transition between those is more striking than common in say structured images. Good examples are smaller tabular data of expression, EHR where DL commonly underperforms. The CV approach without a direct target function to optimize does not scale for optimizing each weight separately.