Data Cleaning & Deduplication

Data cleaning and deduplication tools find records that describe the same person, company or product, merge them into one, and clean the fields around them. Paid tools split three ways: apps that work inside a CRM (HubSpot Data Hub, DemandTools, Insycle, Cloudingo and Plauti for Salesforce, HubSpot or Dynamics 365), list and file cleaners (Data8's per-record bureau, Melissa's verification services, WinPure on your own desktop or server, and Datablist in the browser), and enterprise data quality platforms such as Informatica. Open-source projects cover the same ground for people who can run them: OpenRefine cleans and clusters files without code, Splink, Zingg, dedupe and GoldenMatch deduplicate and link records at scale from Python, SQL or Spark, pyjanitor cleans data frames, and libpostal and zentity are building blocks for address normalization and identity resolution.