Existing regularization methods merely minimize the norms of embeddings to regularize the model, leading to suboptimal performance.

Weempirically assess theimpact oftheseregularization cocktailsforMLPs ina large-scale empirical study comprising 40 tabular datasets and demonstrate that (i) well-regularized plain MLPs significantly outperform recent state-of-the-art specialized neural network architectures, and (ii) they even outperform strong traditionalMLmethods,suchasXGBoost.

We propose a new regularization method to alleviate over-fitting in deep neural networks.

Weconductaseriesofempirical studies showing that overconfidence may not hurt final calibration performance if post-hoc calibration is allowed, rather, the penalty of confident outputs will compress theroom ofpotential improvement inpost-hoc calibration phase.

In this paper,we introduce asimple yet more effectivealternativeto regularize the training inconsistencyinduced bydropout, named asR-Drop. Concretely,ineachmini-batch training, eachdata sample goes through the forward pass twice, and each pass isprocessed by adifferent sub model by randomly dropping out some hidden units.