Two bugs we could plant in the model-key helper without a test noticing

A seeded-fault run on the bring-your-own-key helper caught 6 of 8 mutants; the two escapes, a default-port edge and a private range, are now guarded.

Tools and projects · 1 min read · Source
redteamtestingbyok

We plant eight small faults in platform/llm/byok.mjs and see whether its tests fail. Six did. Two slipped through: an explicit :80 on an Ollama host being rewritten to port 11434, and the 172.16.x to 172.19.x private range being dropped from the "nothing leaves your computer" check. The second matters, because a page only says data stays local when that check is true.

One new test covers both. Re-running the two escaped mutants against it, both are now caught, so the catch rate for this file went from 75% to 100%. No real defects turned up in the helper itself.

More in Tools and projects

All writing · All in this topic · All tools and research