Machine learning tool cleans dirty data -- GCN

Big data is a big deal, but problems within the data can skew results and lead to problematic choices. To help keep data -- and the decisions based on it -- clean, researchers at Columbia University and the University of California at Berkeley have developed new software. ActiveClean analyzes prediction models to determine which mistakes (e.g., typos, outliers and missing values) to edit first, updating the models in the process, according to Columbia. "Big data sets are still mostly combined and edited manually, aided by data-cleaning software like Google Refine and Trifacta or custom scripts developed for specific data-cleaning tasks," university officials said. "The process consumes up to 80 percent of analysts' time as they hunt for dirty data, clean it, retrain their model and repeat the process. Cleaning is largely done by guesswork."