The Idea Behind Transfer Learning: Stand on the Shoulders of Giants

Training big networks on large datasets is expensive considering computational equipment, engineers working on the problem in terms of human resources is also very demanding; trials and errors in training models from the scratch can be time consuming, inefficient and unproductive. Imagine the simple problem of classification on unstructured data in medical domain like sorting the X-rays and training the network to identify if there's broken bone or not. To reach any decent accuracy model has to learn what a broken bone looks like based on images in dataset, it has to make sense of pixels, edges and shapes. This is where the idea of Transfer Learning kicks in: model that is trained on similar data is now taken for the new purpose, weights are frozen and non-trainable layers will be incorporated into a new model that is capable of solving similar problem on smaller dataset. Similarly to Computer Vision type of problem, NLP tasks can also be managed with Transfer Learning methods: for example if we are building a model that takes descriptions of patient symptoms where aim is to predict the possible conditions associated with symptoms; in such case model is required to learn language semantics and how the sequence of words creates the meaning.