Skip to main content
agent-qa on GitHub,904stars
Skip to Content
agent-qa

Write a test

Copy page
View as MarkdownOpen the raw Markdown source.Open in ChatGPTAsk questions about this page.Open in ClaudeAsk questions about this page.Open in MistralAsk questions about this page.Open in VS CodeAsk Copilot Chat about this page.Open in CursorAsk Cursor about this page.

Describe one user journey in plain English, with checks the agent can prove or disprove on screen.

A test is one user journey: a target to run against, and the steps a person would take, written in plain English. agent-qa works out how to do each step on the screen and checks that it happened.

Let your coding agent write it

The quickest way is to describe the flow and let your coding agent use agent-qa's authoring tool. agent_qa_write_test drafts the test, runs it, checks the result is stable and the checks are real, and only then writes the file.

Write and run a test for a flow

Write an agent-qa test for this user flow, run it, and report the result:

<describe the flow, and what must be true at the end>

Read https://vostride.com/docs/agent-qa/guides/first-test.md first.

1. Find the agent-qa target for the app in agent-qa.config.yaml, and any existing tests that already cover part of the flow.
2. Write the test with the agent_qa_write_test MCP tool, passing the flow as the intent. If that tool is not available, write the YAML yourself in the shape the guide shows, leave test-id out, and tell me to open the dashboard's Tests page so it adds the ID.
3. Make every check concrete: name the exact values the screen must show. Never weaken a check to get a pass.
4. Run it with agent_qa_start_run or npx agent-qa run <path>, then read the verdict with agent_qa_get_run_digest or npx agent-qa digest.
5. If it fails, read the failing step's evidence and decide whether the app, the test or the environment is at fault.

Report the test file, the run ID, the verdict, and the evidence for any failure.

The shape of a test

tests/examples/komazon-checkout.yaml
test-id: t_packer-staff-asha-lak-spill-usher-pele-allen-bid-donename: Komazon checkout journeytarget: komazonsteps:  - Add Everyday Ceramic Mug to the bag, then open the shopping bag and verify it holds one Everyday Ceramic Mug.  - Continue to checkout. Sign in with email alex@example.com and password {{secret:KOMAZON_PASSWORD}}, then verify the shipping step is shown.  - Enter street address 24 Cedar Lane, city Portland and postal code 97205. Continue to delivery.  - Select Express delivery and continue with Review order. Verify the review shows one Everyday Ceramic Mug, the entered address, Express delivery and a total of $36.00.  - Place the order using Place order · $36.00. Verify Order confirmed and that the bag is empty.  - Open View orders. Verify Order FW-1001 contains one Everyday Ceramic Mug and the $36.00 total.

test-idA stable ID: t_ followed by ten words. agent-qa's tools create it for you.

Tests live wherever workspace.testMatch points, tests/**/*.yaml by default. The test file reference lists every field.

Writing good steps

  • Write what a person would do. "Open the shopping bag" works better than "click the icon at the top right".
  • Say what must be true. A step that starts with "Verify", "Check" or "Confirm" is a check. Name exact values: "Verify the total is $36.00", not "Verify the total looks right".
  • Keep steps small. One action, or a short sequence followed by one check. A step that fails should tell you where.
  • Never weaken a check to get a pass. If the app is wrong, the test should fail. That is the point.
  • Use variables for data that changes. {{env:ORDER_ID}} reads from .env, --var or a hook. {{secret:PASSWORD}} reads from the secrets file. See Variables and secrets.
  • Give context once. A context: field holds background the agent reads before the first step, such as "The user is already signed in". Never put secrets there.

Test IDs

Every test needs a test-id: t_ followed by ten lowercase words joined by hyphens. agent-qa creates them for you:

  • agent_qa_write_test and the dashboard's test editor write new tests with an ID.
  • Writing YAML by hand? Leave test-id out, start agent-qa dashboard and open Tests. It adds an ID to every test file that lacks one.

Keep the ID when you edit a test. Suites, run history and knowledge refer to it.

Check, then run

npx agent-qa validate tests/checkout.yaml
npx agent-qa run tests/checkout.yaml

validate checks the file against the schema without running anything. run drives the app and ends with a verdict. Name tests explicitly while you work: a bare agent-qa run runs every test in the project, and every test in one run must belong to the same product.

Read the result

A failing step tells you what agent-qa saw, what it tried and why the check failed. Start with the digest, then the step:

npx agent-qa digest
npx agent-qa runs overview <runId>

Review a run covers the evidence. If the verdict is wrong, mark the run so agent-qa learns from it.

Offline replay

Replay the decisions recorded runs made against the current model and score its answers, without changing knowledge or verdicts.

Web testing

Point agent-qa at a web app, choose a browser, and tune timeouts, logs, accessibility checks and recordings.

On this page

Let your coding agent write itThe shape of a testWriting good stepsTest IDsCheck, then runRead the result