Test files
Every field of a test file, the forms a step can take, and the templates steps can use.
A test file describes one journey. Write a test covers how to write good steps; this page lists every field.
test-id: t_ayin-dual-chunk-lou-tablet-ticket-brace-lak-kite-abu
name: Komazon checkout with a shipping address from a hook
target: komazon
context: The shop uses test payment details only.
setup:
- h_phi-arts-vis-band-aira-streng-twig-roi-mino-fold
use:
cache: false
meta:
retries: 1
steps:
- Add Everyday Ceramic Mug to the bag, then open the shopping bag and verify it holds one Everyday Ceramic Mug.
- Continue to checkout. Sign in with email alex@example.com and password {{secret:KOMAZON_PASSWORD}}, then verify the shipping step is shown.
- Type {{env:SHIPPING_ADDRESS}} into the Street address field, enter city Portland and postal code 97205, then continue to delivery.
- Select Express delivery and continue with Review order. Verify the review shows {{env:SHIPPING_ADDRESS}} as the street address.test-id
A stable ID for the test. Suites, run history and knowledge refer to it, so keep it when you edit the test. agent_qa_write_test and the dashboard create IDs, and the dashboard adds one to any test file that lacks it.
- Values
t_followed by ten lowercase words joined by hyphens.- Required
- Required
- Default
- none. agent-qa's tools create it.
name
The name shown in the terminal, the dashboard and reports.
- Values
- any text.
- Required
- Required
- Default
- none.
target
The app to test. The target also decides the platform. When the test runs inside a suite, the suite's target replaces this one.
- Values
- a target name from
registry.targets. - Required
- Required
- Default
- none.
context
Background the agent reads before the first step, such as "The user is already signed in". Never put secrets here.
- Values
- any text.
- Required
- Optional
- Default
- none.
steps
The journey. Each step is plain text, or an object for the rare step that needs more.
- Values
- a list of at least one step.
- Required
- Required
- Default
- none.
A step as text:
steps:
- Open the Orders page and verify the newest order shows a total of $36.00.A step as an object:
steps:
- step: Wait for the import to finish and verify 120 rows are listed.
timeout: 5m
- step: Note the order number shown on the confirmation page.
capture:
variable: ORDER_ID
method: ai
description: the order number on the confirmation page| Field | Meaning |
|---|---|
step | The step's text. Required. |
timeout | A time limit for this step, overriding use.timeout.step. |
capture.variable | Saves a value from the screen as a variable, read later as {{env:NAME}}. |
capture.method | ai reads the value described in description; regex takes the first group of pattern from the accessibility tree; selector reads the element selector matches. |
setup and teardown
Hooks to run before the first step and after the last. See Hooks.
- Values
- lists of hook IDs from your hooks file.
- Required
- Optional
- Default
- none.
use
Overrides the project's run settings for this test. Anything left out is inherited from the suite, then the project.
| Key | Meaning |
|---|---|
llm | A model entry from registry.llms. |
browser | name, headless, viewport, and remote: a remote from registry.remotes, or local. See Remote browsers and devices. |
timeout | step, test and navigation. |
planner | maxActions, maxPlanningTurns and previousStepCount. |
logCapture | console and network. |
cache | false turns off the action cache for this test. |
authState | A saved login: a name, or { name, load, capture }. |
device | A device from registry.devices. Without it, the test runs on its target's device. |
mobile.appState | preserve or reset. |
parallel | false keeps the whole run sequential. |
workers | How many tests run at once when parallel. |
meta
| Key | Meaning |
|---|---|
timeout | A time limit for the whole test, overriding use.timeout.test. |
retries | How many times to run the test again after a failure. Only the last attempt counts. A pass after a retry is classed as flaky. |
record | true or false to record a video of this test, overriding --record and the project setting. |
Templates in steps
| Template | Becomes |
|---|---|
{{env:NAME}} | A variable from .env, --var, a hook or a capture. |
{{secret:NAME}} | A secret from your secrets file, typed at the moment of the action and masked everywhere else. |
{{runHook:"h_…"}} | Runs a hook before the step. |
{{runJS:"code"}} | The result of running JavaScript in the page, on web. |
{{appium: command, key: value}} | Runs an Appium command, on mobile. |
A bare {{NAME}} is a validation error. A template with no value fails its step before it acts, with a message naming where the value could come from.
More than one test in a file
Separate tests with ---. Each needs its own test-id.
Not part of a test
platform: it comes from the target.maxAttemptson a step,use.healinganduse.planner.maxSubActions: removed in v0.2. Recovery is built into every step. See Self-healing.retriesandscreenshoton a step: removed. Setmeta.retriesto run the whole test again, and pass--screenshot-mode every-steptoagent-qa runfor a screenshot after every step.