The Importance Of No Free Lunch Theorems In Deep Learning

"The no free lunch theorem calls for prudency when solving ML problems by requiring that you test multiple algorithms and solutions with a clear mind and without prejudice." In a paper titled, 'The Lack of A Priori Distinctions Between Learning Algorithms', that dates back to 1996, David Wolpert explored the following questions: He showed that for any two algorithms, A and B, there are as many scenarios where A will perform worse than B as there are instances where A will outperform B. In short, for all possible problems, average performance of both the algorithms is the same. Although the no free lunch theorem by Wolpert has a more theoretical than practical appeal, there are some implications that should still be taken into account by everyone working with machine learning algorithms. These theorems prove that under a uniform distribution over search problems or learning problems, all algorithms perform equally. Search and learning are key aspects of ML and the NFL theorems have something to deliver here.