Using AI to Clean Up Big Data - DZone Big Data

Big data is a hot topic right now, but the successful utilization of that data largely rests on the ability of organizations to provide clean, accurate and usable data to employees to make real-time insights. Suffice to say, much of the data held in organizational databases is anything but clean, and few organizations seem willing to undertake the laborious job of cleaning it up. AI may be about to come to the rescue, as a team of researchers from Columbia University and the University of California at Berkeley has developed some automated software to do the job for you. The software, called ActiveClean, uses prediction models to test out datasets, and uses the results to understand the fields that require cleaning whilst simultaneously updating the models at the same time. As with so many laborious processes, human error can be a significant factor, so ActiveClean takes them out of the equation in two of the most error prone areas: finding the dirty data to begin with, and then updating models accordingly.