Test

A test YAML file describes one user journey. It selects a target from agent-qa.config.yaml, gives the agent context, optionally runs hooks, and lists the steps the agent must execute and verify.

This page uses a web app example because most agent-qa projects start with browser testing.

Example web test

test-id: t_vog-earing-wap-git-tim-assert-teras-mill-aus-ila
name: Create urgent task and finish it
target: issue-tracker-web
use:
  browser:
    name: chromium
    headless: true
    viewport:
      width: 1280
      height: 720
  cache: false
  timeout:
    step: 3m
    test: 30m
context: |
  The web app is available at the URL configured by the issue-tracker-web target.
  The user is already signed in to the QA workspace.
  Task data comes from the workspace .env file.
steps:
  - Open the Tasks page.
  - Click the create task button.
  - In the task title field, enter exactly "{{env:TASK_TITLE}}".
  - In the task description field, enter exactly "{{env:TASK_DESCRIPTION}}".
  - Set the status to "{{env:STATUS_IN_PROGRESS}}".
  - Set the priority to "{{env:TASK_PRIORITY}}".
  - Submit the task.
  - Verify task "{{env:TASK_TITLE}}" appears in the Created view.
  - Run the API verification hook {{runHook:"h_decode-ouf-laid-remain-icing-iao-vang-bur-hem-vira"}}.
  - Verify TASK_FOUND is exactly "true"; the current value is "{{env:TASK_FOUND}}".
  - Open task "{{env:TASK_TITLE}}" and change the status to "{{env:STATUS_DONE}}".
  - Verify the task no longer appears in the Assigned view.

test-id

test-id: t_vog-earing-wap-git-tim-assert-teras-mill-aus-ila

Description: Generated stable ID for the test.

Possible values: canonical test IDs that start with t_ and contain 10 id-agent words.

Required: yes.

Default: none. Generate it with agent-qa ids generate test or the dashboard.

name

name: Create urgent task and finish it

Description: Human-readable test name shown in the dashboard, CLI output, and run artifacts.

Possible values: any string.

Required: yes.

Default: none.

target

target: issue-tracker-web

Description: Target name from registry.targets in agent-qa.config.yaml.

Possible values: any configured target name. For web tests, point this at a target whose platform is web.

Required: yes.

Default: none.

context

context: |
  The web app is available at the URL configured by the issue-tracker-web target.
  The user is already signed in to the QA workspace.

Description: Background passed to the agent before the first step.

Possible values: any string, usually a multi-line block.

Required: no.

Default: not set.

Do not put secrets in context. Use {{secret:NAME}} or hook environment variables for credentials.

setup

setup:
  - h_seed-task-workspace-bird-lake-slate-palm-cloud-frost

Description: Hook IDs that run before the test steps.

Possible values: hook IDs registered in hooks.yaml.

Required: no.

Default: no setup hooks.

teardown

teardown:
  - h_update-borg-artha-any-packet-derive-torch-front-plied-bed

Description: Hook IDs that run after the test finishes.

Possible values: hook IDs registered in hooks.yaml.

Required: no.

Default: no teardown hooks.

use

use:
  browser:
    name: chromium
    headless: true
  cache: false

Description: Per-test overrides for supported global and suite defaults. Use authState here for web tests that should start already signed in.

Possible values: the UseOverrideSchema shape: browser, timeout, healing, planner, logCapture, cache, authState, mobile, llm, parallel, and device.

Required: no.

Default: inherited from suite, global config, and CLI flags.

use.browser.name

use:
  browser:
    name: chromium

Description: Browser engine for this web test.

Possible values: chromium, firefox, or webkit.

Default: inherited.

use.browser.headless

use:
  browser:
    headless: true

Description: Whether this test runs without a visible browser window.

Possible values: true or false.

Default: inherited.

use.browser.viewport

use:
  browser:
    viewport:
      width: 1280
      height: 720

Description: Browser viewport override for this test.

Possible values: numeric width and height.

Default: inherited.

use.timeout.step

use:
  timeout:
    step: 3m

Description: Timeout for one step.

Possible values: duration strings such as 30s, 3m, and 1h.

Default: inherited.

use.timeout.test

use:
  timeout:
    test: 30m

Description: Timeout for the whole test.

Possible values: duration strings.

Default: inherited.

use.timeout.navigation

use:
  timeout:
    navigation: 90s

Description: Timeout for navigation actions.

Possible values: duration strings.

Default: inherited.

use.healing.maxAttempts

use:
  healing:
    maxAttempts: 1

Description: Self-healing retry budget for this test.

Possible values: number.

Default: inherited.

use.planner.maxSubActions

use:
  planner:
    maxSubActions: 8

Description: Planner sub-action budget for this test.

Possible values: number.

Default: inherited.

use.planner.previousStepCount

use:
  planner:
    previousStepCount: 4

Description: Previous-step context count for this test.

Possible values: number.

Default: inherited.

use.logCapture.console

use:
  logCapture:
    console: true

Description: Captures browser console logs during this test.

Possible values: true or false.

Default: inherited.

use.logCapture.network

use:
  logCapture:
    network: true

Description: Captures browser network logs during this test.

Possible values: true or false.

Default: inherited.

use.cache

use:
  cache: false

Description: Enables or disables action cache for this test.

Possible values: true or false.

Default: inherited.

use.authState

use:
  authState: qa-admin

Description: Named web auth state to load before creating the browser context. The name is resolved for the selected target; tests cannot reference arbitrary auth-state file paths.

Possible values: a configured lowercase auth-state name.

Default: not set.

For capture, hook access, security, and mobile boundaries, see Auth state.

use.llm

use:
  llm: codex

Description: LLM config name to use for this test.

Possible values: a name from registry.llms.

Default: inherited.

use.parallel

use:
  parallel: false

Description: Allows parallel execution where supported.

Possible values: true or false.

Default: inherited.

use.device

use:
  device: android-local

Description: Mobile device profile override. Web tests usually do not set this.

Possible values: a name from registry.devices.

Default: inherited or CLI-selected.

use.mobile.appState

use:
  mobile:
    appState: preserve

Description: Native mobile app-state override. Web tests do not use this value.

Possible values: preserve or reset.

Default: inherited.

meta

meta:
  timeout: 20m
  retries: 1
  record: true

Description: Test-level execution metadata.

Possible values: timeout, retries, and record.

Required: no.

Default: not set.

meta.timeout

meta:
  timeout: 20m

Description: Test timeout metadata.

Possible values: duration strings.

Default: not set.

meta.retries

meta:
  retries: 1

Description: Retry count metadata.

Possible values: number.

Default: not set.

meta.record

meta:
  record: true

Description: Recording metadata for the test.

Possible values: true or false.

Default: not set.

steps

steps:
  - Open the Tasks page.
  - Click the create task button.

Description: Ordered instructions for the agent. A test must contain at least one step.

Possible values: string steps or structured step objects.

Required: yes.

Default: none.

string step

steps:
  - Open the dashboard and verify the project switcher is visible.

Description: A plain natural-language instruction.

structured step

steps:
  - step: Search for task "{{env:TASK_TITLE}}" and open the first matching result.
    timeout: 90s
    retries: 1
    screenshot: true
    maxAttempts: 2

Description: A step object with per-step controls.

step.step

steps:
  - step: Search for task "{{env:TASK_TITLE}}".

Description: The natural-language instruction for a structured step.

Required: yes for structured steps.

step.timeout

steps:
  - step: Wait for the import status to finish.
    timeout: 90s

Description: Timeout for this step.

Possible values: duration strings.

Default: inherited step timeout.

step.retries

steps:
  - step: Click retryable sync.
    retries: 1

Description: Retry count for this step.

Possible values: number.

Default: not set.

step.screenshot

steps:
  - step: Verify the task detail page is open.
    screenshot: true

Description: Requests a screenshot around this step.

Possible values: true or false.

Default: not set.

step.maxAttempts

steps:
  - step: Save the task.
    maxAttempts: 2

Description: Step-level max attempts.

Possible values: number.

Default: not set.

capture

steps:
  - step: Copy the task key from the detail header.
    capture:
      variable: TASK_KEY
      method: regex
      pattern: "TASK-[0-9]+"
  - Verify the activity feed mentions "{{env:TASK_KEY}}".

Description: Captures runtime data from a structured step for later {{env:NAME}} interpolation.

capture.variable

capture:
  variable: TASK_KEY

Description: Variable name written into the run environment.

Required: yes.

capture.method

capture:
  method: regex

Description: Capture strategy.

Possible values: regex, selector, or ai.

Required: yes.

capture.pattern

capture:
  method: regex
  pattern: "TASK-[0-9]+"

Description: Regex pattern used by regex capture.

Required: when the regex method needs a pattern.

capture.selector

capture:
  method: selector
  selector: "[data-testid='task-key']"

Description: Selector or stable element reference used by selector capture.

Required: when the selector method needs a selector.

capture.description

capture:
  method: ai
  description: Extract the task key from the task detail header.

Description: Semantic extraction instruction used by ai capture.

Required: when the AI method needs semantic guidance.

Inline hooks

steps:
  - Run the API verification hook {{runHook:"h_decode-ouf-laid-remain-icing-iao-vang-bur-hem-vira"}}.
  - Verify TASK_FOUND is exactly "true"; the current value is "{{env:TASK_FOUND}}".

Description: Inline hooks run at a specific point in the step list. They run before variable interpolation for that step, and exported variables are available to later steps.

Possible values: {{runHook:"h_..."}} with a hook ID registered in hooks.yaml.

Use captures, hooks, or setVariable actions for runtime data. The older variables block is not part of the current test schema.