Reviews: Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples

The paper proposes a novel way of sampling (or weighing) data-points during training of neural networks. The idea is, that one would like to sample data-point more often which could be potentially classified well but are hard to learn (in contrast to outliers or wrongly labeled ones). To find' them the authors propose two (four if split into sampling and weighing) schemes: The first one (SGD-*PV) proposes to weigh data-points according to the variance of the predictive probability of the true label plus its confidence interval under the assumption that the prediction probability is Gaussian distributed. The second one (SGD-*TC), as far as I understand, encodes if the probability of choosing the correct label given past prediction probabilities is close to the decision threshold. The statistics needed (means and variances of p) can be computed on-the-fly during a burn-in phase of the optimizer; they can be obtained from a forward pass of the network which is computed anyways.