Eval corpus
An optional set of recorded decisions with known answers, used to measure the agent and to hold new lessons to replay and canary checks.
Most projects never need this page. An eval corpus is a reviewed set of recorded decisions with known right answers. With one, agent-qa can measure the agent itself, and it holds new knowledge to stronger checks before trusting it.
Where it lives
The corpus is the agent-qa-eval/ folder next to agent-qa.config.yaml. Its files are written once and never overwritten, and you commit them. Recordings it points to stay out of git.
The folder appears the first time you mark a run: every mark saves the marked decision as an example with a known answer. init never creates it.
What changes once it exists
Canary checks start right away. After each run, agent-qa replays a few decisions that were given an unproven record, once with the record and once without, to see whether it helped. This happens at most four times per run (--max-ablations or eval.gate2.maxAblations), and each replay calls your model about five times. It cannot be switched off while the folder exists.
Replay checks start once a noise floor is measured. From then on, a new fact or caution is promoted only after it passes a replay check and a canary check, instead of the default verdict floor.
The checks
| Check | What it does |
|---|---|
| Replay | Asks the model again on recorded screens from the corpus, with and without the candidate's line in the brief. The candidate must fix the decision it is about and change nothing else, by more than the measured noise. |
| Canary | Serves the candidate, marked UNVERIFIED, on later real runs and records whether it helped or hurt the steps it was given to. |
| Your mark | A person's judgment. It promotes on its own and always wins. |
A check's result is stored with the record in evidence/gates.jsonl. npx agent-qa knowledge gates <recordId> shows it, and agent-qa doctor reports the current learning mode.
Commands
| Command | What it does | Calls the model |
|---|---|---|
agent-qa eval extract | Turns recorded runs into staged decision recordings. | No |
agent-qa eval seed | Commits recordings into the corpus. The first seed of a product needs --held-out-fraction and --private-fraction. | No |
agent-qa eval census | Summarizes the corpus per product. | No |
agent-qa eval floor | Measures the noise floor by replaying the corpus several times. | Yes |
agent-qa eval replay | Asks the current model again on recorded decisions and scores the answers. | Yes |
agent-qa eval report | Builds a report from a replay. --commit --baseline makes it the baseline. | No |
agent-qa eval gate | Compares a fresh replay with the baseline. Exits 0 for no regression, 1 for a regression and 2 when it cannot decide. | Yes |
agent-qa eval replay-gate <candidateId> | Runs the replay check for one candidate and records the result. | Yes |
agent-qa eval calibrate | Measures how reliable each kind of label is against your marks. | No |
agent-qa eval adversarial | Stages a batch of deliberately misleading variants of real cases for review. | Sometimes |
A typical order: run tests and mark them, then extract, seed, census, floor --commit, replay --source committed, report --source committed --commit --baseline, and from then on gate before a release.
Judges
Some comparisons need a second model to decide whether two answers mean the same thing. Name one with eval.judge, or up to three with eval.judges, using entries from registry.llms:
eval:
judges: [judge-a, judge-b, judge-c]Each judge must come from a different model family than the model under test and than each other. agent-qa never picks a judge for you. A panel decides by strict majority.
Label calibration
agent-qa eval calibrate measures how reliable each kind of automatic label is against your marks. From that it sets how much weight recovered steps carry as examples, and how far a new record is trusted at first. It needs at least 20 marked examples per kind of label and calls no model. It previews by default, and --commit publishes the result. Restart a running dashboard or daemon afterwards so it reads the new values. Until a calibration is published, agent-qa doctor reports trust priors: provisional.
If calibration is refused
When a published calibration cannot be read, agent-qa refuses to continue, and agent-qa doctor fails its learning check. Inspect agent-qa-eval/labels/calibration.json and the revisions it references, without changing them. Restore the exact committed files from git or a trusted backup, then run agent-qa doctor again. Never delete those files or write a new calibration by hand to get past the refusal.
To publish a different calibration on purpose, preview agent-qa eval calibrate, review it, then run agent-qa eval calibrate --commit --replace.