The problem as far as I can see is that far too few papers use a basic and simple baseline to compare against as a control. They are always comparing against the state of the art and previous DL techniques, but rarely to do they include basic correlation analysis, linear / logistic regressions, etc., as a basis for comparison. In statistics one doesn't just say "we got X performance which was better than Y performance", one says "we show that the effect size is better than control by X amount, and confirm that this actually represents an improvement and is not likely a bias induced by random sampling of the data with 95% confidence." But DL papers often just include final test set performance and traces of loss function per iteration, and say, look X learns faster than Y and Z and ends up with less error. Often this is even done without confidence intervals, which, for methods that depend on random initial conditions, is a sin.