Flaky tests and quarantine
How agent-qa tells flaky and environmental failures from real ones, and sets repeat offenders aside so they stop failing your run.
A test that fails because the network blinked should not keep your run red, and it should not teach agent-qa anything about your app. agent-qa diagnoses every failure first, and sets aside tests that keep failing for reasons outside your app.
Root causes
After each run, analysis gives every failing test a root cause:
| Root cause | Meaning |
|---|---|
app_bug | The app did the wrong thing. |
test_bug | The test asked for the wrong thing. |
flake | It fails some of the time, for example it passed when tried again. |
environment | Something around the app broke: a server, a network, a device. |
timeout | The test ran out of time. |
config_error | The configuration is wrong. |
llm_error | The model call failed. |
unknown | Not enough evidence to say. |
agent-qa digest <runId> shows the root causes for a run, grouped by failure.
When a test is quarantined
Only flake, environment and timeout failures can quarantine a test. Bugs in the app or the test never do; they stay red until fixed.
agent-qa looks at the last five runs of the product. When the same failure, meaning the same element, error and root cause, shows up in two of them, the test is flagged. In three, it is quarantined. A few extra guards apply when a verdict came from rules rather than the model: a flaky verdict needs a pass and a fail on the same build, environment needs the model's verdict, and at most two tests are quarantined automatically per run.
While a test is quarantined
The test still runs and its result is still reported. Its failure no longer fails the run:
- the terminal marks it as quarantined and not counted, and the summary counts quarantined failures;
- JUnit adds a
quarantinedproperty to the test case; - the run's exit code follows the other tests.
Leaving quarantine
A test leaves quarantine on its own after five passing runs in a row. An entry also expires after 30 days; if the test is still failing then, it comes back.
The ledger
Quarantine lives in agent-qa-knowledge/quarantine.jsonl, an append-only log you commit with the rest of your knowledge. Every entry, renewal and release is a line, so the history is reviewable.
Manual control
Quarantine has no dedicated command. Use the function layer, or ask in chat:
npx agent-qa fn quarantine.list
npx agent-qa fn quarantine.add --input '{"product":"komazon","filePath":"tests/checkout.yaml","testName":"Komazon checkout journey","owner":"payments-team"}'
npx agent-qa fn quarantine.release --input '{"product":"komazon","filePath":"tests/checkout.yaml","testName":"Komazon checkout journey"}'A test is identified by its product, its file path from the project folder and its name. quarantine.add on a test that is already quarantined renews the entry and records the owner. expiresAt sets a different expiry than 30 days.
Marks on flaky failures
Marking a run whose failure was diagnosed as flaky, environmental or a timeout teaches nothing, because there is nothing about your app to correct. The receipt says quarantined with the reason. That mark does not quarantine the test itself.