Configuration

agent-qa is designed to keep your QA system under complete team control. Configuration, tests, suites, hooks, custom rules, local bindings, auth metadata, and memory are all represented as files, so teams can review, version, and collaborate on QA changes the same way they review application code.

The dashboard can create and edit those files for you, but the files remain the source of truth. You can change config toggles in the UI, write a test by hand, update a suite in a pull request, or let an AI coding agent propose a diff. The next run reads the same files.

Global Configuration

Define workspace files, dashboard and MCP services, LLM configs, targets, devices, auth plugins, analytics, and default run behavior.

Test

Author one natural-language journey with generated IDs, target selection, hook references, variables, captures, and per-test overrides.

Suite

Group tests into an ordered workflow with shared context, shared hooks, target selection, and suite-level execution overrides.

Hook

Register sandboxed scripts in hooks.yaml, then implement Node.js, Bun, Python, or Bash hooks that export runtime variables.

Variables & Secrets

Load non-secret variables from .env, keep sensitive values in the secrets file, and pass temporary data between hooks and steps.

Local Device Config

Keep machine-specific device serials, local app paths, and provider credentials out of the portable global config.

Agent Rules

Tune the QA agent with project-specific execution rules without changing every test file.

auth.json

Understand where saved LLM credentials live, what the credential schema looks like, and how to create it at CI runtime.

File map

Most teams start with this set of files:

agent-qa.config.yaml       # global workspace, services, registry, plugins, and use defaults
tests/**/*.yaml            # single natural-language test definitions
suites/**/*.suite.yaml     # ordered groups of tests
hooks.yaml                 # hook registry
scripts/*.mjs              # hook implementation files
.env                       # non-secret variables used by steps and hooks
.env.secrets.local         # secret values used through {{secret:NAME}}
agent-qa.local.yaml        # local device, app, and provider bindings
agent-rules.md             # additional QA agent instructions
~/.agent-qa/auth.json      # saved LLM credentials, outside the project by default

Generated runtime data also remains file-backed. Dashboard state, artifacts, cache, and memory live under the configured paths, usually inside .agent-qa/ or agent-qa-memory/.

Configuration layers

agent-qa resolves configuration from a small set of explicit layers:

agent-qa.config.yaml defines project-wide defaults.
Suite YAML can override supported use settings for every test in the suite.
Test YAML can override supported use settings for that test.
CLI flags such as --device, --no-cache, --no-memory, --headless, --no-headless, and --var KEY=VALUE apply to the current command.

Use the dashboard when you want a guided editor. Use file edits when you want reviewable diffs. Both paths update the same source files.