Scaling up with Distributed Tensorflow on Spark – Towards Data Science

As you may have experienced in the past, or probably will at some point, running out of memory is a very common issue in Data Science. Due to the large facets of businesses, it is not uncommon to create datasets with over 10,000 or more features. We may choose to process such dataset with tree-algorithms. Deeplearning, however, easily engineers features more automatically and process them into a model of your choice. An often occurring problem is figuring how to train your favorite model within a respectful amount while processing such a huge amount of data.