One walk of the tree instead of 2,161

Our research homepage builder walked its 456 MB source tree 2,161 times per build. One shared walk cut build time 20% with byte-identical output.

The research homepage at /research/ is generated by a Python script that reads 1,080 research projects. Each project card shows how many tests it has, and the index shows how many projects need nothing beyond the Python standard library.

Both numbers came from Path.rglob. To decide which imports are "ours", the script walked all of research/ once. Then, for every project, it walked that project's directory twice more: once for *.py, once for test*.py. That is 2,161 directory walks over 18,605 files on every build. The build runs three times in its own test module, again in every ventures-tests CI job, and again in every site deploy.

A profile put the walks at about a third of the build. The fix walks the tree once with os.scandir, keeps every file's relative path per project, and answers both questions from that list.

Two checks guarded it. The first: the site built before and after the change is byte-identical (diff -r). The second: a new test compares the shared walk with the rglob calls it replaced, on a small tree that includes hidden folders, __pycache__, an upper-case .PY file and a project that does not exist.

The result is about 0.5 s per build, 2.60 s down to 2.08 s on average. It is a small saving, but it repeats in every build. As in the earlier passes, the cost was an algorithm (asking the same filesystem question 2,161 times), not the language.

Full reading, profile and ranked next steps: docs/history/ci-perf-2026-11-17-research-tree-one-walk.md.

All writing · More in Research findings · All tools and research