Reviews: Cold Case: The Lost MNIST Digits

The authors introduce a new version of the MNIST data set that they call QMNIST, which is the result of a very thoughtful and systematic analysis of existing materials used to build the original MNIST. I am impressed by the meticulous investigation carried out to recover the precise processing steps needed to generate MNIST examples from the original NIST images. The QMNIST data set is thus the product of a very accurate reconstruction of the original process (though the authors note that some minor discrepancies are still present). The authors then investigate whether the performance of popular classification methods measured using the new QMNIST test set actually differ from that measured on the original MNIST test set. Overall, I think this research is well conducted and presented in a very clear way.