Why transfer learning works or fails?

During the NIPS tutorial talk given in 2016, Andrew Ng said that transfer learning -- a subarea of machine learning where the model is learned and then deployed in related, yet different, areas -- will be the next driver of machine learning commercial success in the years to come. This statement would be hard to contest as avoiding learning large-scale models from scratch would significantly reduce the high computational and annotation efforts required for it and save data science practitioners lots of time, energy, and, ultimately, money. As an illustration of these latter words, consider Facebook's DeepFace algorithm that was the first to achieve a near-human performance in face verification back in 2014. The neural network behind it was trained on 4.4 million labeled faces -- an overwhelming amount of data that had to be collected, annotated, and then trained on for 3 full days without taking into account the time needed for fine-tuning. It won't be an exaggeration to say that most of the companies and research teams without Facebook's resources and deep learning engineers would have to put in months or even years of work to complete such a feat, with most of this time spent on collecting an annotated sample large enough to build such an accurate classifier.