It is quite obvious that ML teams developing new models or algorithms expect that the performance of the model on test data will be optimal. But many times that just doesn't happen. The above list is not exhaustive though. In this article, we'll discuss the process which can solve multiple above-mentioned problems and ML teams be very mindful while executing it. It is widely accepted in the machine learning community that preprocessing data is an important step in the ML workflow and it can improve the performance of the model. "A study by Bezdek et al. (1984) found that preprocessing the data improved the accuracy of several clustering algorithms by up to 50%." "A study by Chollet (2018) found that data preprocessing techniques such as data normalization and data augmentation can improve the performance of deep learning models."

This screencast shows how to use Skymind's DataVec to ingest Comma Separated Values from a text file, convert the fields to numeric using a DataVec Transform Process in Spark, and save the modified data. Transforming non-numeric data to numeric data is a key preliminary step to using a Neural Network to analyze the data.