Questions & Intuition for Tackling Deep Learning Problems

This question is particularly relevant for supervised training problems. The typical premise underlying such problems is that a small high-quality dataset (say N entities) can help your model approximate an underlying function, which can generalize to your entire dataset (1000N entities). The allure of these approaches, of course, is that humans do the hard work on a small amount of data, and machines learn to replicate the work for a wider range of examples. In the real world though, problems don't always have an underlying pattern that can be identified. Humans draw on external general knowledge to solve cognitive challenges more often than we realize, which often leads us to falsely expect our algorithms to be able to solve the same challenges, without the benefit of the general knowledge that we posses.