dupehound is the repository's copy-paste detector, written in Rust. Pointed at the 11,023 .mjs files under ventures/, it reports 39,263 duplicated blocks, covering 38.4% of 1.44 million lines. That number is true, and it gives you nothing to work on. Nobody reads 39,263 rows.
The list is that long because a clone detector reports pairs. A block pasted into 1,000 files shows up as 999 pairs, and those rows are spread through the list between everything else.
Families
The new --families option folds the pairs into families. A family is one block plus every place it was pasted. The join rule is simple. A clone links its two sides. Inside a single file, two spans link when they share at least half of the shorter span's tokens. Plain overlap was too loose: the end of one pasted function touching the start of the next was enough to chain unrelated blocks into one huge family. The detector also pairs every copy with the block's first occurrence, so all N copies point at one place and meet there.
For each family the report gives:
- how many copies there are, and how many match token for token;
- the shortest and longest copy, in lines. A wide spread means the copies have drifted apart;
- whether every copy sits in a file with the same name. A thousand
usage.mjsfiles is a template; - the deepest directory that holds every copy, which is the natural place for a shared module;
- a removable lines range. The low end is the larger of two counts: every copy minus one, times the shortest copy; or the identical copies minus one, times their length. The high end is every copy's lines except the longest copy's. The range does not subtract the import and call lines a shared module would add.
On this repository, the 39,263 pairs become 3,214 families. 1,950 of them are a single pair. Only 224 have ten or more copies, and those 224 hold 337,000 of the 394,000 lines in the low estimate. The plain scan and the family scan both take 1.1 to 1.3 seconds and about 575 MB, so grouping adds nothing measurable (benchmark, "Families").
What the top of the list says
| Copies | Identical | Lines | Removable | Same file name |
|---|---|---|---|---|
| 1,025 | 1,006 | 10 to 76 | 76,380 to 77,060 | usage.mjs |
| 123 | 121 | 16 to 482 | 57,840 to 57,872 | no |
| 124 | 121 | 12 to 457 | 54,840 to 54,878 | no |
| 137 | 37 | 7 to 498 | 16,704 to 33,888 | no |
| 180 | 180 | 82 | 14,678 | usage.test.mjs |
At first glance, the first three rows are 190,000 lines of debt. They aren't. Each venture here is published as its own directory, so it cannot import a file from elsewhere in the tree. A ../../platform/... import would 404 on the live site, so shared code is copied in on purpose. The copies are also checked. platform/proposal/sync.test.mjs fails if any venture's proposal.mjs differs by one byte from the source copy. platform/ci/usage_contract.mjs runs every usage.mjs against a fake browser store and checks how it behaves.
So the most useful thing the families view showed was not a refactor. It showed what the report still lacks: a way to mark which families something already guards. Once those are marked, the top of the list is the copies nobody is watching. The fourth row, 137 copies with only 37 identical, is the kind to read first. Its copies range from 7 to 498 lines, and nothing here says that spread was intended.
Try it
dupehound --families src/
dupehound --families --json src/ > families.json
The browser page now shows a families table under the pair list, worked out by the same engine compiled to WebAssembly. A test checks that the page's JSON matches the CLI's exactly. PMD's CPD also lists every occurrence of a match. The removable-lines ranking, the identical counts and the template hint are new here.
Next is a --guarded list, so families that a sync test already pins drop out of the ranking.