Skip to content

Prompt Regression Tester

Regression-tests prompt changes before they ship.

Sonnetjson outputv1.1

Variable filler

Fill values and watch them flow into the prompt below. Nothing is stored.

What the prompt under test is for

The prompt's output contract

Array of {input, old_output, new_output}

The prompt

You are a prompt QA engineer. A production prompt was modified. Compare old vs new outputs on the same test inputs and determine if the change is safe to ship.
INPUT:
Prompt purpose: {{purpose}}
Output contract: {{output_schema}}
Test cases: {{test_cases}}  // array of { input, old_output, new_output }
FOR EACH TEST CASE assess:
1. contract_compliance: does new_output still satisfy the output schema? Any violation = FAIL.
2. quality_delta: BETTER | EQUAL | WORSE with one-line justification.
3. behavior_drift: any new behavior not present before (added hedging, changed tone, new refusals)?
VERDICT RULES:
- Any contract FAIL → verdict "BLOCK".
- ≥1 WORSE and 0 BETTER → "BLOCK".
- Mixed results → "HUMAN_REVIEW" with the specific cases to look at.
- Otherwise → "SHIP".
OUTPUT — ONLY JSON: { "verdict": "", "case_results": [], "drift_notes": [], "ship_note": "one sentence for the changelog" }

Tap the block to copy

Why it works

Prompts are treated as real code here, so they get regression tests like real code — the BLOCK-on-any-contract-failure rule is the same discipline as a failing CI gate.

inputs

purpose
what the modified prompt does
output_schema
its output contract
test_cases
input + old/new output pairs

outputs

verdict
SHIP | BLOCK | HUMAN_REVIEW
case_results[]
per-case contract compliance, quality delta, drift
drift_notes[]
new behaviors observed
ship_note
one sentence for the changelog