Your file
What each column holds
Guessed from the headers. Set a column to Ignore to leave it out, for example a record ID or notes.
| Column | First value | Compare as |
|---|
Comparing rows…
Results
Duplicate groups
Borderline pairs
What each field was worth in this file
Weights in bits, learned from your file: +7 means agreement at that level is about 27 = 128 times likelier between duplicates than between different people. Blank cells count for nothing either way.
Who this is for
Small businesses and agencies merging contact lists from two systems, sales teams cleaning a CRM before a migration, nonprofits with a donor list that has grown three entries for the same family, bookkeepers with a customer file full of near-identical names. Exact-match dedupe (the button in most spreadsheet and email tools) only removes rows that are spelled identically. On the Febrl benchmark below it finds about a third of the real duplicates.
A worked example
Press Load the contact example: 21 placeholder rows. The tool groups Maria Okafor with María Okafor (accent, upper-case email), Robert with Bob Lindqvist (same phone, written with +1), Priya Raman with Priya Ramen (typo, different email, same phone digits), and Thomas with Tom Grant (an email +tag). It holds back Grace and Jim Whitfield, who share a surname, company and town but not a first name, email or phone, and lists them for you to decide.
When to use something else
Over 50,000 rows, ongoing matching inside a CRM, or matching two different files against each other on a schedule: a data-quality product or the open-source Python libraries Splink and recordlinkage handle those. To fix an email list's formatting before an import, use the email list CSV checker.
How it works
Blocking. Comparing every pair of 5,000 rows is 12.5 million comparisons, so only rows that share a key are compared: the same email, the same last seven phone digits, a similar-sounding surname (Soundex) with the same first initial, the same first letters of a name plus postcode, the same street and number, and a few more.
Comparison. Each field of a candidate pair gets an agreement level: names after accents are dropped and common nicknames joined (Bob and Robert), scored with the Jaro-Winkler string similarity; emails with case, Gmail dots and +tags ignored; phones on their last 10 digits; companies without Inc, LLC or Ltd; addresses with St and Street joined.
Weights from your file. The Fellegi-Sunter model gives each agreement level a weight, log2(m/u): m is how often duplicates agree at that level, u how often different people do. Both are learned from your own rows by the EM algorithm, so a surname shared by half the file counts for less than a rare one would in a file where surnames rarely repeat. The weights add up to a match probability for each pair; pairs above the threshold are joined, and pairs that share a member are grouped transitively.
Merging. In each group the row with the most filled cells is kept, and its blank cells are filled from the other rows (the most common value). Columns where the rows disagree are named so you can check them.
Accuracy on a benchmark with known answers
The Febrl data sets are standard record-linkage benchmarks: synthetic Australian person records (name, address, date of birth, ID number) with typos, missing values, swapped fields and nickname variants, where the true duplicates are known. At the default 90% setting:
| Set | Rows | True duplicate pairs | Found (recall) | Correct (precision) | Exact match on name and postcode, found |
|---|---|---|---|---|---|
| Febrl 1 | 1,000 | 500 | 99.8% | 100% | 36.2% |
| Febrl 2 | 5,000 | 1,934 | 99.6% | 100% | 29.1% |
| Febrl 3 | 5,000 | 6,538 | 99.8% | 100% | 25.9% |
With only name and address columns (no date of birth or ID number, closer to a typical contact list), it found 81 to 85% of the true pairs at 99.7 to 100% precision, and held 54 to 799 pairs back for review because the first names differed. Switching that guard off found 95 to 97% at 99.8 to 100% precision on these sets; on a business contact list, where colleagues and relatives share a surname and company, the guard is the safer default. A benchmark is not your file: check the groups. Full figures and the script are in the README.
Method and citation
Fellegi and Sunter (1969), "A Theory for Record Linkage", JASA 64(328); Winkler (1988), "Using the EM Algorithm for Weight Computation in the Fellegi-Sunter Model of Record Linkage"; Jaro (1989), JASA 84(406); Winkler (1990), "String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage". Written from the papers' mathematics; no code was ported.
@article{fellegi1969,
author = {Fellegi, Ivan P. and Sunter, Alan B.},
title = {A Theory for Record Linkage},
journal = {Journal of the American Statistical Association},
volume = {64}, number = {328}, pages = {1183--1210}, year = {1969},
doi = {10.1080/01621459.1969.10501049}
}
@inproceedings{winkler1988,
author = {Winkler, William E.},
title = {Using the {EM} Algorithm for Weight Computation in the {Fellegi-Sunter} Model of Record Linkage},
booktitle = {Proceedings of the Section on Survey Research Methods, American Statistical Association},
pages = {667--671}, year = {1988}
}
Febrl data set 1 is from the Febrl project (Peter Christen, Australian National University) as redistributed in the BSD-licensed recordlinkage 0.16 package; its people are generated, not real.
Questions
How do I find duplicate contacts that are not exact matches?
Exact-match dedupe only catches rows that are spelled the same. This tool compares every likely pair field by field with graded agreement (exact, very close, close, different), weighs each field by how much it tells you in your own file, and groups rows whose combined match probability passes a threshold you set. It catches Bob and Robert Lindqvist at the same phone number, Priya Raman and Priya Ramen at the same company, and the same person with a Gmail address written with and without dots.
How accurate is it?
On the three Febrl deduplication benchmark sets (1,000 to 5,000 synthetic person records with typos, missing values and swapped fields, where the true duplicates are known), it found 99.6 to 99.8% of true duplicate pairs with 100% precision at the default setting. An exact match on first name, last name and postcode found 26 to 36%. Your file is not a benchmark: check the groups before you delete anything.
Is my data uploaded?
No. The file is read and matched in your browser tab. Nothing is sent anywhere, and there is no account.
Will it merge two different people?
It can, so it shows you why each pair matched and lists borderline pairs separately. By default, two rows with clearly different first names are never merged automatically unless an email, phone, ID number or date also agrees exactly; colleagues and family members who share a surname, company and town go to the review list instead.
How many rows can it handle?
Up to 50,000 rows. It compares only pairs that share a blocking key (same email, same last 7 phone digits, similar-sounding surname with the same first initial, and so on), so 5,000 rows take about half a second.
Not decided
Which of two conflicting values is right (an old and a new address both look valid to it), whether merging records is allowed under your own data-protection duties, how frequent a name is in the population (it learns only from your file), and matching across languages or scripts. Common nicknames are English only.