Natural language testing that is a file format, not a prompt
An agent-qa test says what should happen and what proves it happened. The runtime reads the live interface and works out how to get there, so no step is pinned to a selector. The sentences sit inside a schema: a test ID, a target, context, hooks, and interpolated values, in YAML your team reviews like any other file.
A step names the behavior, not the element
Each step is one sentence. It names a control the way a person would say it out loud, and it names the outcome that should follow: 'Click New issue.' 'Verify the Issues table contains a row titled Fix mobile checkout.' The file never says which node that is, because it does not have to. agent-qa resolves the sentence against the live interface using visible roles, labels, and screen state, at the moment the step runs.
Nothing is compiled in between. There is no recording and no generated script sitting between your app and the test, so there is no second artifact to keep in sync with the first. The sentence is the part you maintain, and it is also the part a reviewer reads.
The structure around the sentences
Plain English sounds imprecise until you see the rest of the file. A test declares a test-id, a name, and a target, plus optional context that tells the run what it is looking at before step one. Hooks run before the steps, after them, or at one exact point in the middle, in a sandboxed container. Values are interpolated rather than typed in: a URL arrives as {{env:LINEAR_URL}}, a credential as {{secret:API_TOKEN}}, and a fixture a hook just created arrives the same way.
Because it is a schema, it can be checked. agent-qa validate reads your config, your test files, and the suite references between them before a browser ever opens. What lands in the diff is a contract: the behavior in the steps, and everything that has to be exact in the fields around them.
test-id: t_checkout-smoke
name: Guest checkout completes
target: storefront-web
context: |
The storefront runs at the URL configured by the storefront-web target.
Product data is seeded by the workspace setup hook.
steps:
- Open the home page and search for "espresso grinder".
- Open the first search result.
- Add the product to the cart.
- Start checkout as a guest.
- Fill the shipping form with the workspace test address.
- Place the order.
- Verify the confirmation page shows an order number.The same file, whoever wrote it
A product manager can write the steps for a flow they specified. An engineer can add the target and the hooks. A QA engineer can tighten the assertion. A coding agent can write the whole file from the context it already holds about your product. The output is identical in each case: reviewable YAML in the repository, opened in a pull request beside the feature it covers.
That works because there is no authoring tool to learn. No recorder, no vendor editor, no machine-only format. You create a file, run it with one command, and read the run back in a local dashboard. Anyone who can describe a user journey can add coverage, and everyone else can review what they added.
# initialize a workspace
npx agent-qa init
# run a test
npx agent-qa run tests/checkout-smoke.yaml
# inspect runs in the local dashboard
agent-qa dashboard --port 3470 --openFrequently asked questions
What is natural language testing?
Natural language testing writes each step as a sentence about the product rather than as code against the page. In agent-qa a step names a control a user can see and the outcome that should follow, and the runtime resolves it against the live interface using visible roles, labels, and screen state. The tests are YAML files in your repository.
Plain English sounds flaky. What keeps a run deterministic?
The structure around the sentences. Each test declares a test-id, a target, and the context a run starts with. Data arrives by interpolation, so {{env:LINEAR_URL}} and {{secret:API_TOKEN}} resolve from your environment instead of being typed into a step. Hooks seed and verify state through your own code. agent-qa validate checks the file against the schema before anything runs.
What is actually in an agent-qa test file?
test-id, name, and target are required. context is an optional block that tells the run what it is looking at. setup and teardown list hook IDs, and a hook can also be called inline at one exact step. use carries per-test overrides such as browser, cache, and timeouts. steps is the ordered list of natural-language sentences. The test file reference documents every field.
How do URLs, credentials, and test data get into a test?
As variables, never as literals in a step. Stable values live in .env, secrets live in .env.secrets.local, and temporary values come from hook output merged into the active run. Steps reference them as {{env:NAME}} and {{secret:NAME}}, so one file runs against a local environment, a preview, or CI without being edited.
Who on the team writes these tests?
Anyone who can describe the journey. A product manager, an engineer, and a QA engineer each write in the language they would use to explain the flow out loud, and a coding agent produces the same file from the context it already has. Whoever typed it, the artifact is the same YAML, reviewed in the same pull request.