Comparing Machine Learning Models: Statistical vs. Practical Significance

A lot of work has been done on building and tuning ML models, but a natural question that eventually comes up after all that hard work is -- how do we actually compare the models we've built? If we're facing a choice between models A and B, which one is the winner and why? Could the models be combined together so that optimal performance is achieved? A very shallow approach would be to compare the overall accuracy on the test set, say, model A's accuracy is 94% vs. model B's accuracy is 95%, and blindly conclude that B won the race. In fact, there is so much more than the overall accuracy to investigate and more facts to consider.