Did the new model break your prompts?

Did the new model or prompt break your outputs?

Paste your saved outputs from before and after a model swap or prompt edit (one JSON object per line with an id and an output). Get a per-case list of what broke: invalid JSON, lost keys, new refusals, truncation, lost numbers, lost formatting.

Nothing is uploaded and no model is called. It compares text you already have. It is a tripwire for changes, not a judge of quality.

1. Your outputs

Baseline (before) — JSONL or JSON array

Candidate (after) — JSONL or JSON array

Flag length changes beyond (x)

Flag word overlap below (0–1)

Allowed regressed cases (CI gate)

Gate it in CI

node cli.mjs diff baseline.jsonl candidate.jsonl
node cli.mjs diff baseline.jsonl candidate.jsonl --max-regressions 2 --fail-on warn
node cli.mjs diff baseline.jsonl candidate.jsonl --json

Get cli.mjs and engine.mjs — Node 18+, nothing to install, no network call. Exit 1 when regressions exceed the limit. More: How to regression-test prompts when switching LLM models.

Common questions

Does this call any model or API?

No. It compares two files of outputs you already saved. It makes no network call, needs no API key and never sees your prompts unless you put them in the output text.

What does it actually check?

Per case id: empty or missing output, JSON that stopped parsing or lost keys or changed a value's type, new refusal or apology wording, unclosed code fences, large length changes, numbers that disappeared, lost lists, headings or code blocks, truncation, and low word overlap. Errors are likely to break a consumer; warnings deserve a human look.

Is this a quality score for a model?

No. It is a tripwire for changes, not a judge of which output is better. A flagged case can be an improvement and a clean case can still be worse; read the flagged ones and keep your own evaluation for quality.

Is my data uploaded anywhere?

No. The page works in your browser and sends nothing. The CLI reads local files only.