How knowledge is learned
How a finished run becomes records, which checks each change passes, and when a candidate is promoted.
After a run reaches its verdict, agent-qa turns what the run proved into knowledge. The verdict is already final at that point, so learning can never turn a green run red.
After the verdict
Three jobs run in order, inside a 60-second budget:
- Analysis gives every failing test a root cause: an app bug, a test bug, a flaky failure, an environment or timeout problem, a config or model error, or unknown. Rules handle most failures. The model is asked only about the ones the rules cannot place.
- Curation turns the run's evidence into knowledge, as described below.
- Proposals run extra checks on new candidates when an eval corpus exists. Without one, this job has nothing to do.
Work that does not finish in time stays queued, and the next local run picks it up. agent-qa run --detach skips the wait and leaves all of it for later.
What becomes a candidate
Candidates come from the evidence by fixed rules. No model is involved at this point.
| From the run | Candidate |
|---|---|
| A step that passed its check and acted on exactly one element | A fact: navigational when the step moved to another page, selector when the element's locator is known, behavioral otherwise |
| A value a step reported, or the actual value of a passing check | A measurement fact |
| A test whose root cause was a timeout, a flaky failure or the environment | A caution |
| A test of two or more steps where every step passed its check | A procedure, written only when a model is available |
A run yields at most 50 candidates, and identical candidates merge. Secrets from your secrets file are masked before a candidate is queued.
The curator
For each candidate, the curator first searches for records about the same anchors. Then the run's own model writes exactly one change:
| Change | What it does |
|---|---|
Add | Creates a new record. |
Confirm | Adds a confirmation to an existing record. |
Supersede | Replaces a record that is no longer true with a new one. |
Deprecate | Retires a record. |
MintUnknown | Raises an open question instead of guessing. |
Noop | Changes nothing. |
What the app showed is passed to the curator as untrusted data, never as instructions. Without a model, or when the model's answer is unusable, the curator falls back to fixed templates and can only add, confirm or supersede. Those records are reworded the next time a model curates the same candidate.
Checks
Before a change is written, code checks it:
- every record it cites exists, and its anchors agree with the record it changes;
- it supersedes at most one record per anchor in one pass;
- agent-qa cannot retire an active record on its own; only a person can;
- if the record changed in the meantime, the change is written again against the new version.
A change that fails a check is not applied as written. It is kept as a proposal instead.
The promotion gate
Every new record starts as a candidate. On a local run, the promotion gate runs straight away and decides whether the candidate becomes active. Only facts and cautions can be promoted. A candidate needs:
- confirmed evidence from a verified step, and complete evidence;
- for a fact, exactly one matching element on screen; for a caution, a root cause;
- no person having deprecated it, and its anchor not paused for churn (see below).
A candidate that passes is promoted along one of these paths, from weakest to strongest:
| Path | When it applies |
|---|---|
deterministic-verdict-floor | The default. The evidence above is enough. |
gate1-lower-trust, gate1+gate2 | With an eval corpus, the candidate must also pass a replay check and a live canary check. |
user-sovereign | A person confirmed it, for example by marking a run. This always wins. |
A candidate that does not pass stays a candidate with a named reason, such as no-confirmed-evidence or needs-enrichment. Procedures never become active; they are served by their own matcher once later runs confirm them.
Local runs and CI
agent-qa detects CI from the CI variable and the usual vendor variables, such as GITHUB_ACTIONS or GITLAB_CI.
| Local run | CI run | |
|---|---|---|
| New knowledge | Written and promoted when it passes the gate | Written as candidates only |
| A correction to a record | Supersedes it | Saved as a separate proposal; the original is untouched |
| Retiring a record | Allowed when the checks agree | Becomes an open question |
Your own corrections, such as run marks and accepting, rejecting or forgetting a record, apply the same way everywhere. agent-qa never commits; whatever a CI job learns stays in that workspace unless the job commits it.
Change over time
- Superseding. When the app changes, a newer record supersedes the old one, and both stay on disk.
- Measurements. A numeric value builds a baseline over five runs. A clear, sustained shift after that supersedes it.
- Churn guard. If one anchor is superseded more than three times within five runs, memidex raises an unknown and stops serving facts about that anchor until it has been stable for five runs.
- Burst guard. Eight or more new anchors in one scope within five runs also raise an unknown.
What you will see
After the verdict, the run prints a few lines on stderr:
curation: 3 learned change(s) applied, 0 kept proposed (local runs apply learned knowledge to the knowledge root)
curation: added 2, confirmed 1, superseded 0, deprecated 0, proposed 0In CI the first line reads recorded … (ci runs never promote learned knowledge) instead. Each promotion or hold is reported too, such as curation: <id> promoted via deterministic-verdict-floor or curation: <id> held: no-confirmed-evidence.