How a run works

A test is a target and a list of plain-English steps. A run takes one or more tests, drives the app step by step, and ends with a verdict for every test and one for the whole run.

One step at a timeCopy Anchor Link

For each step, agent-qa repeats a short loop until the step is done:

1. ObserveCopy Anchor Link

   It reads the current screen: the accessibility tree and page structure, plus a screenshot when the model can see images.

2. RecallCopy Anchor Link

   It reads the knowledge brief for this test: your policies, cautions about known failure modes, and facts learned on earlier runs.

3. Plan and actCopy Anchor Link

   The model picks the next action, such as a click, a typed value or a navigation, and agent-qa performs it on the real app.

4. VerifyCopy Anchor Link

   It checks the result against what the step asked for. A step that says "Verify the total is $36.00" passes only when the screen proves it.

A step may take several actions. Each step is capped at 30 actions and 30 model calls by default (use.planner.maxActions and use.planner.maxPlanningTurns), and stops with the limit as its reason when it needs more.

Two things can skip the model:

• The action cache. When the same step meets the same screen, agent-qa replays the action that completed it last time. See Caching.
• Procedures. A flow that memidex has seen pass several times can replay without planning. See How knowledge reaches a run.

When an action fails or the check rejects its result, the failure goes back to the model, which plans a recovery. A test that passes after such a recovery is healed. See Self-healing.

VerdictsCopy Anchor Link

Each test ends with one of these:

The run's verdict is failed if any test failed or timed out, cancelled if the run was stopped, healed if any test healed, and passed otherwise. A quarantined test's failure is reported but does not fail the run.

Exit codesCopy Anchor Link

Suites and parallel runsCopy Anchor Link

A suite runs its tests one after another on one browser or device. When one fails, the rest of that suite is skipped.

Tests named directly run one at a time by default. With use.parallel: true or --parallel, up to use.workers tests run at once (3 by default).

Every test in one run must belong to the same product. A target's product is its product field, or the target's own name. Tests on different products need separate agent-qa run invocations.

After the verdictCopy Anchor Link

The verdict and the exit code are final the moment the run ends. agent-qa then spends up to 60 seconds on post-run work:

1. Analysis. It classifies each failure, for example as an app bug, a test bug, a flaky failure or an environment problem.
2. Curation. It turns what the run proved into knowledge records. See How knowledge is learned.
3. Proposals. Extra checks on new knowledge, when an eval corpus exists.

A line on stderr reports each stage as pending, running, complete, failed or skipped. Work that does not finish in time stays queued, and the next local run finishes it. agent-qa run --detach skips the wait entirely. Daemon and background work has the details.

Where a run livesCopy Anchor Link

Each run gets an ID such as r_ followed by ten words, and a folder under .agent-qa/runs/. Read runs through agent-qa rather than from the folder:

npx agent-qa runs list
npx agent-qa runs overview <runId>
npx agent-qa digest <runId>

Review a run walks through the evidence, and the dashboard shows the same runs in a browser.

The modelCopy Anchor Link

One model entry, use.llm, plans and checks every step of a run. A test or suite can name a different entry in its own use.llm. The same model writes up analysis and knowledge after the run.

Models that accept images get a screenshot with each decision. Set vision: false on a model entry to run text-only: agent-qa then works from the accessibility tree and page structure, and cannot check purely visual details.