Switching model or editing a prompt changes outputs in ways unit tests do not see. Save before and after outputs, then run the diff tool over them.
Models differ in how closely they follow format instructions, how readily they refuse, how long they write and where they stop. A prompt tuned against one model can start returning prose around your JSON, drop a field, or end mid-sentence on another, and none of that shows up in a unit test of the surrounding code.
Run your real prompts over a fixed set of inputs on the current model, and save each result with a stable id as JSONL. Keep 20 to 100 cases that cover your hardest formats. Re-run the same inputs on the candidate model or prompt and save those the same way, then compare the two files.
Run the CLI in your workflow after the candidate run and let its exit code decide: node cli.mjs diff baseline.jsonl candidate.jsonl --max-regressions 0. Exit 1 means regressions exceeded the limit; use --fail-on warn to be stricter.
That the candidate is as good. These are deterministic text checks. They catch format and contract breaks cheaply, and they leave factual accuracy, tone and safety to your own evaluation and review.
- run: node cli.mjs diff baseline.jsonl candidate.jsonl --max-regressions 0