--- title: "Eval corpus" description: "The agent-qa eval corpus: recorded planner decisions with known answers, replay and canary checks for new knowledge, judges, noise floors and the eval commands." canonical_url: "https://vostride.com/docs/agent-qa/self-improvement/evals" md_url: "https://vostride.com/docs/agent-qa/self-improvement/evals.md" last_updated: "2026-10-24T01:12:08+05:30" --- > [Documentation index](https://vostride.com/docs.md) ยท [Site index](https://vostride.com/llms.txt) > Follow the Markdown links to read individual pages; the indexes list the available documentation. # Eval corpus > The agent-qa eval corpus: recorded planner decisions with known answers, replay and canary checks for new knowledge, judges, noise floors and the eval commands. Advanced Most projects never need this page. An eval corpus is a reviewed set of recorded decisions with known right answers. With one, agent-qa can measure the agent itself, and it holds new knowledge to stronger checks before trusting it. ## Where it lives The corpus is the `agent-qa-eval/` folder next to `agent-qa.config.yaml`. Its files are written once and never overwritten, and you commit them. Recordings it points to stay out of git. The folder appears the first time you [mark a run](https://vostride.com/docs/agent-qa/self-improvement/feedback.md): every mark saves the marked decision as an example with a known answer. `init` never creates it. ## What changes once it exists **Canary checks start right away.** After each run, agent-qa replays a few decisions that were given an unproven record, once with the record and once without, to see whether it helped. This happens at most four times per run (`--max-ablations` or `eval.gate2.maxAblations`), and each replay calls your model about five times. It cannot be switched off while the folder exists. **Replay checks start once a noise floor is measured.** From then on, a new fact or caution is promoted only after it passes a replay check and a canary check, instead of the default [verdict floor](https://vostride.com/docs/agent-qa/memory/learning.md#the-promotion-gate). ## The checks | Check | What it does | | --- | --- | | Replay | Asks the model again on recorded screens from the corpus, with and without the candidate's line in the brief. The candidate must fix the decision it is about and change nothing else, by more than the measured noise. | | Canary | Serves the candidate, marked `UNVERIFIED`, on later real runs and records whether it helped or hurt the steps it was given to. | | Your mark | A person's judgment. It promotes on its own and always wins. | A check's result is stored with the record in `evidence/gates.jsonl`. `npx agent-qa knowledge gates ` shows it, and `agent-qa doctor` reports the current learning mode. ## Commands | Command | What it does | Calls the model | | --- | --- | --- | | `agent-qa eval extract` | Turns recorded runs into staged decision recordings. | No | | `agent-qa eval seed` | Commits recordings into the corpus. The first seed of a product needs `--held-out-fraction` and `--private-fraction`. | No | | `agent-qa eval census` | Summarizes the corpus per product. | No | | `agent-qa eval floor` | Measures the noise floor by replaying the corpus several times. | Yes | | `agent-qa eval replay` | Asks the current model again on recorded decisions and scores the answers. | Yes | | `agent-qa eval report` | Builds a report from a replay. `--commit --baseline` makes it the baseline. | No | | `agent-qa eval gate` | Compares a fresh replay with the baseline. Exits 0 for no regression, 1 for a regression and 2 when it cannot decide. | Yes | | `agent-qa eval replay-gate ` | Runs the replay check for one candidate and records the result. | Yes | | `agent-qa eval calibrate` | Measures how reliable each kind of label is against your marks. | No | | `agent-qa eval adversarial` | Stages a batch of deliberately misleading variants of real cases for review. | Sometimes | A typical order: run tests and mark them, then `extract`, `seed`, `census`, `floor --commit`, `replay --source committed`, `report --source committed --commit --baseline`, and from then on `gate` before a release. ## Judges Some comparisons need a second model to decide whether two answers mean the same thing. Name one with `eval.judge`, or up to three with `eval.judges`, using entries from `registry.llms`: ```yaml eval: judges: [judge-a, judge-b, judge-c] ``` Each judge must come from a different model family than the model under test and than each other. agent-qa never picks a judge for you. A panel decides by strict majority. ## Label calibration `agent-qa eval calibrate` measures how reliable each kind of automatic label is against your marks. From that it sets how much weight recovered steps carry as examples, and how far a new record is trusted at first. It needs at least 20 marked examples per kind of label and calls no model. It previews by default, and `--commit` publishes the result. Restart a running dashboard or daemon afterwards so it reads the new values. Until a calibration is published, `agent-qa doctor` reports `trust priors: provisional`. ### If calibration is refused When a published calibration cannot be read, agent-qa refuses to continue, and `agent-qa doctor` fails its learning check. Inspect `agent-qa-eval/labels/calibration.json` and the revisions it references, without changing them. Restore the exact committed files from git or a trusted backup, then run `agent-qa doctor` again. Never delete those files or write a new calibration by hand to get past the refusal. To publish a different calibration on purpose, preview `agent-qa eval calibrate`, review it, then run `agent-qa eval calibrate --commit --replace`. These commands exist for teams that maintain a corpus, and their options may change between releases. Run `agent-qa eval --help` for the current flags. ## Documentation navigation - [All documentation](https://vostride.com/docs.md) - Parent: [Self-improvement](https://vostride.com/docs/agent-qa/self-improvement.md) - Previous: [Flaky tests and quarantine](https://vostride.com/docs/agent-qa/self-improvement/quarantine.md) - Next: [Write a test](https://vostride.com/docs/agent-qa/guides/first-test.md)