FUZZY DEDUPE · RUNS IN YOUR BROWSER · NOTHING UPLOADED

Find the duplicates an exact match misses

For anyone cleaning a contact list, CRM export, customer file or mailing list before an import or a mail-out: paste or open a CSV and get groups of rows that are probably the same person or business, even when one says Bob and the other Robert, one has a typo in the surname, or the email is written two ways. Each group comes with the reason it matched, and you download a merged file with one row per person.

Statistical matching on the file you give it. Not legal, compliance or professional advice; check groups before deleting anything. The contact example is invented placeholder data, not any business's customers.

Your file

Who this is for

Small businesses and agencies merging contact lists from two systems, sales teams cleaning a CRM before a migration, nonprofits with a donor list that has grown three entries for the same family, bookkeepers with a customer file full of near-identical names. Exact-match dedupe (the button in most spreadsheet and email tools) only removes rows that are spelled identically. On the Febrl benchmark below it finds about a third of the real duplicates.

A worked example

Press Load the contact example: 21 placeholder rows. The tool groups Maria Okafor with María Okafor (accent, upper-case email), Robert with Bob Lindqvist (same phone, written with +1), Priya Raman with Priya Ramen (typo, different email, same phone digits), and Thomas with Tom Grant (an email +tag). It holds back Grace and Jim Whitfield, who share a surname, company and town but not a first name, email or phone, and lists them for you to decide.

When to use something else

Over 50,000 rows, ongoing matching inside a CRM, or matching two different files against each other on a schedule: a data-quality product or the open-source Python libraries Splink and recordlinkage handle those. To fix an email list's formatting before an import, use the email list CSV checker.

How it works

Blocking. Comparing every pair of 5,000 rows is 12.5 million comparisons, so only rows that share a key are compared: the same email, the same last seven phone digits, a similar-sounding surname (Soundex) with the same first initial, the same first letters of a name plus postcode, the same street and number, and a few more.

Comparison. Each field of a candidate pair gets an agreement level: names after accents are dropped and common nicknames joined (Bob and Robert), scored with the Jaro-Winkler string similarity; emails with case, Gmail dots and +tags ignored; phones on their last 10 digits; companies without Inc, LLC or Ltd; addresses with St and Street joined.

Weights from your file. The Fellegi-Sunter model gives each agreement level a weight, log2(m/u): m is how often duplicates agree at that level, u how often different people do. Both are learned from your own rows by the EM algorithm, so a surname shared by half the file counts for less than a rare one would in a file where surnames rarely repeat. The weights add up to a match probability for each pair; pairs above the threshold are joined, and pairs that share a member are grouped transitively.

Merging. In each group the row with the most filled cells is kept, and its blank cells are filled from the other rows (the most common value). Columns where the rows disagree are named so you can check them.

Accuracy on a benchmark with known answers

The Febrl data sets are standard record-linkage benchmarks: synthetic Australian person records (name, address, date of birth, ID number) with typos, missing values, swapped fields and nickname variants, where the true duplicates are known. At the default 90% setting:

SetRowsTrue duplicate pairsFound (recall)Correct (precision)Exact match on name and postcode, found
Febrl 11,00050099.8%100%36.2%
Febrl 25,0001,93499.6%100%29.1%
Febrl 35,0006,53899.8%100%25.9%

With only name and address columns (no date of birth or ID number, closer to a typical contact list), it found 81 to 85% of the true pairs at 99.7 to 100% precision, and held 54 to 799 pairs back for review because the first names differed. Switching that guard off found 95 to 97% at 99.8 to 100% precision on these sets; on a business contact list, where colleagues and relatives share a surname and company, the guard is the safer default. A benchmark is not your file: check the groups. Full figures and the script are in the README.

Method and citation

Fellegi and Sunter (1969), "A Theory for Record Linkage", JASA 64(328); Winkler (1988), "Using the EM Algorithm for Weight Computation in the Fellegi-Sunter Model of Record Linkage"; Jaro (1989), JASA 84(406); Winkler (1990), "String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage". Written from the papers' mathematics; no code was ported.

@article{fellegi1969,
  author = {Fellegi, Ivan P. and Sunter, Alan B.},
  title = {A Theory for Record Linkage},
  journal = {Journal of the American Statistical Association},
  volume = {64}, number = {328}, pages = {1183--1210}, year = {1969},
  doi = {10.1080/01621459.1969.10501049}
}
@inproceedings{winkler1988,
  author = {Winkler, William E.},
  title = {Using the {EM} Algorithm for Weight Computation in the {Fellegi-Sunter} Model of Record Linkage},
  booktitle = {Proceedings of the Section on Survey Research Methods, American Statistical Association},
  pages = {667--671}, year = {1988}
}

Febrl data set 1 is from the Febrl project (Peter Christen, Australian National University) as redistributed in the BSD-licensed recordlinkage 0.16 package; its people are generated, not real.

Questions

How do I find duplicate contacts that are not exact matches?

Exact-match dedupe only catches rows that are spelled the same. This tool compares every likely pair field by field with graded agreement (exact, very close, close, different), weighs each field by how much it tells you in your own file, and groups rows whose combined match probability passes a threshold you set. It catches Bob and Robert Lindqvist at the same phone number, Priya Raman and Priya Ramen at the same company, and the same person with a Gmail address written with and without dots.

How accurate is it?

On the three Febrl deduplication benchmark sets (1,000 to 5,000 synthetic person records with typos, missing values and swapped fields, where the true duplicates are known), it found 99.6 to 99.8% of true duplicate pairs with 100% precision at the default setting. An exact match on first name, last name and postcode found 26 to 36%. Your file is not a benchmark: check the groups before you delete anything.

Is my data uploaded?

No. The file is read and matched in your browser tab. Nothing is sent anywhere, and there is no account.

Will it merge two different people?

It can, so it shows you why each pair matched and lists borderline pairs separately. By default, two rows with clearly different first names are never merged automatically unless an email, phone, ID number or date also agrees exactly; colleagues and family members who share a surname, company and town go to the review list instead.

How many rows can it handle?

Up to 50,000 rows. It compares only pairs that share a blocking key (same email, same last 7 phone digits, similar-sounding surname with the same first initial, and so on), so 5,000 rows take about half a second.

Not decided

Which of two conflicting values is right (an old and a new address both look valid to it), whether merging records is allowed under your own data-protection duties, how frequent a name is in the population (it learns only from your file), and matching across languages or scripts. Common nicknames are English only.