Self-improvement
How agent-qa gets better at testing your app with every run, what it checks before it trusts a lesson, and how your feedback steers it.
agent-qa gets better at testing your app with every run. It does not rewrite its own prompts or code. It improves what it knows about your app, what it can replay without asking the model, which tests it can trust, and, when you tell it a verdict was wrong, the knowledge or the test behind that verdict.
The loop
Run
A test runs and every step records its evidence. The verdict and the exit code are final before any learning starts.
Diagnose
Analysis gives every failure a root cause. A flaky failure or a broken environment is treated as such, not as a lesson about your app.
Learn
What the run proved becomes candidate records in memidex: facts, cautions and procedures. The model words them; code decides what is allowed.
Check
A candidate is promoted only with confirmed evidence from a verified step. With an eval corpus, it must also pass a replay check and a live canary check.
Use
The next run gets the relevant records in its brief. Each later run adds evidence for or against them, so trust rises and falls with what the app actually does.
Correct
When a verdict is wrong, mark the run. agent-qa fixes the record or the test behind it in the same call.
What improves
| What | How it improves | Where it lives |
|---|---|---|
| Knowledge about your app | Facts, cautions and procedures learned from runs and marks. | agent-qa-knowledge/memory/ |
| Speed and cost | Actions that completed a step replay without the model on the same screen. | .agent-qa/cache/ |
| Trust in your suite | Tests that keep failing for flaky or environmental reasons are quarantined. | agent-qa-knowledge/quarantine.jsonl |
| Your tests | Changed only when a mark points at the test itself, or when you ask agent-qa to write or edit one. | tests/ |
| New tests it writes | Your best-running tests become examples for the next test agent-qa writes. | Derived on your machine |
What it never does
- Change a verdict or exit code after the run ends.
- Edit its own prompts or code.
- Edit your policies. Only you do that.
- Make git commits. Every change waits for your review.
- Promote anything learned in CI.
Local runs and CI
Local runs apply what they learn: candidates that pass the gate are promoted at once. CI runs only propose, so a CI job never changes what your team's runs trust. Corrections you make yourself, such as marks, apply in both. How knowledge is learned has the details.
You stay in control
| You want to | Do this |
|---|---|
| See what changed | Review git diff -- agent-qa-knowledge, or run agent-qa knowledge diff <ref>. |
| Fix a wrong verdict | agent-qa runs mark <runId> --kind false-fail, false-pass or confirmed-bug |
| Drop a record | Reject or forget it with agent_qa_change_knowledge |
| Undo a batch | agent-qa knowledge rollback --job <jobId> |
| Release a quarantined test | agent-qa fn quarantine.release |
What it costs
Learning uses your configured model, after the verdict and within a minute:
- Analysis calls the model only for failures the rules cannot classify.
- The curator makes one call per candidate, and a run yields at most 50.
- A mark may make one short call to word its fix.
With an eval corpus, replay and canary checks make more calls. Eval corpus explains when and how many.