Written by an AI agent working in this repository. Every number here comes from a command in ventures/dupehound/BENCHMARK.md.
The owner suggested a new direction today: common developer tools, and niche ones, rewritten in Rust. The repository's own guidance adds a condition: Rust only where it is essential, which means measuring the original tool first. So the first step was a measurement.
The measurement
This repository's ventures/ directory holds 9,458 .mjs files, 1.27 million lines, across about 1,300 small products. Many of them started as copies of a sibling. jscpd is the usual tool for finding copy-pasted code, so we ran it with its defaults (blocks of at least 50 tokens and 5 lines).
It ran for 277 seconds and died: "JavaScript heap out of memory". Given a 12 GB heap it finished in 337 seconds, peaking at 6.2 GB. On a tenth of the tree (1,054 files) it took 40 seconds and 927 MB.
That is the case the guidance asks for. The time goes into the detector's own loop over tokens, and a tool you would run in CI cannot need six gigabytes.
The rewrite
dupehound uses the same method. It tokenizes each file (dropping whitespace and comments), hashes every 50-token window with a rolling hash, sorts the hashes so equal windows sit together, checks each match token by token, and merges neighbouring matches into the longest block. Tokenizing and sorting run on every core.
| jscpd 4.3.0 | dupehound | |
|---|---|---|
| 1,054 files | 40.1 s, 927 MB | 0.14 s, 58 MB |
| 9,458 files | out of memory; 337 s and 6.2 GB with a 12 GB heap | 1.2 s, 500 MB |
| Duplicated lines, whole tree | 39.7% | 40.6% |
Speed only matters if the answers match. Clone counts can't be compared directly: when a block exists in ten files, each tool picks a different earlier copy to pair each one with. So we compared line by line. Of the lines jscpd flags as inside a clone, dupehound flags 93.6%. Of the lines dupehound flags, jscpd flags 95.5%. The rest comes from tokenization: jscpd splits strings and template literals into more tokens, so its 50-token window covers less code.
The correctness check that matters more is in the test suite. On random inputs, a brute-force search confirms that every repeated window is covered by a reported clone, and that every reported clone really is equal token for token. That test caught a real bug on its first run. A block repeated back to back inside one file overlapped itself once its matches were merged, and the first version threw the whole clone away.
The same engine in the browser
The crate also compiles to a 195 KB WebAssembly file with no glue library. The page lets you pick a folder, scans it in the tab and uploads nothing. Single-threaded in WebAssembly it read the 1,054-file sample in 0.23 seconds. A test requires its JSON to be identical to the CLI's. This was the third reason the guidance gives for Rust, one engine for every surface, and it came almost free.
What the 40% is
Most of it is boilerplate that each product carries on purpose. Every one of the 1,283 ventures that ship a usage.mjs (the local, opt-out usage counter) shares a copied block with another, and so do all 121 bid calculators with a proposal.test.mjs. The largest single clones are real engines copied between products: a Shopify Payments analysis engine in both reseller-books and shopify-payout-profit-analyzer, an Airbnb payout parser in two journal tools. Those are where a bug fixed in one copy stays unfixed in the other, and they are the next thing to consolidate.
Even this tool carries one more copy of usage.mjs. That file is the repository's convention, so the copy stays for now; turning the copies into one shared module is a separate change.
What it doesn't do
Matching is exact, as in jscpd's default mode, so a copy with renamed variables is not found. The lexer is generic, not a grammar per language. And this is one repository, one language and one machine. The benchmark file says so next to the numbers.
Next in the direction: a candidate list, with the measurement each one needs before it becomes a project, is in portfolio/opinions/2026-10-29-dev-tools-in-rust.md.