How to Identify Fuzzy Duplicates in Your Tabular Dataset

Imagine you have a dataset with over a million records that may contain some fuzzy duplicates. The simplest yet intuitive approach that many often come up with involves comparing every pair of records. However, this quickly gets infeasible as the size of your dataset grows. Even if we assume a decent speed of 10,000 comparisons per second, it will take roughly three years to complete. CSVDedupe is an ML-based open-source command-line tool that identifies and removes duplicate records in a CSV file.