Sometimes more data can hurt!

On a recent blog post I've discussed a scalable sparse linear regression model I've developed at work. One of it's interesting properties is that it's an interpolating model – meaning it has 0-training error. This is because it's over parameterized and thus can fit the training data perfectly. While 0-training error is usually associated with over-fiting, the model seems to perform pretty well on the test set. Reports of hugely over-parameterized models that seem to not suffer from overfiting (especially in deep learning) have been accumulating in recent years and so the literature on subject.