Natural language testing, without giving up source control
Plain English is the most durable way to express what a test should verify — it describes the user, not the implementation. agent-qa makes English the executable contract: natural-language YAML files, versioned in git, run by a model-driven runtime on web and mobile.
Intent is the durable part of a test
Selectors churn. Frameworks get replaced. 'Sign in, add an item to the cart, verify the total' stays true for the life of the product. Natural-language testing keeps the durable part — intent — as the source of truth and delegates the volatile part — concrete actions — to a runtime that re-derives them against the live app on every run.
Unlike no-code platforms that trap English tests in a vendor UI, agent-qa keeps them as files. They diff cleanly, review like code, and belong to your repository — which means product engineers, QA specialists, and coding agents all author and maintain them in the same workflow.
Structured enough to trust
Natural language doesn't mean vague. Tests declare a target, context, environment variables, hooks, and ordered steps; templated values like {{env:TEST_EMAIL}} keep runs deterministic; suites compose tests into ordered plans. The result reads like a checklist and executes like a harness.
test-id: t_login-mobile
name: Sign in on the mobile app
target: shop-app-android
steps:
- Launch the app.
- Dismiss the onboarding carousel if it appears.
- Open the profile tab and choose "Sign in".
- Enter the email "{{env:TEST_EMAIL}}" and password "{{env:TEST_PASSWORD}}".
- Submit the form.
- Verify the profile tab shows the account's display name.English tests that compound
Each run adds behavioral observations to file-backed memory and reuses cached action plans where the app hasn't changed. Over weeks, the suite converges: faster runs, fewer wrong turns, richer context for every new test. That's the difference between natural-language testing as a demo and as an engineering practice.
# initialize a workspace
npx agent-qa init
# run a test
npx agent-qa run tests/checkout-smoke.yaml
# inspect runs in the local dashboard
agent-qa dashboard --port 3470 --openFrequently asked questions
What is natural language testing?
Natural language testing expresses test steps as human-readable intent — 'open the tasks page, create a task, verify it appears' — and relies on a runtime to translate intent into concrete actions. agent-qa implements this with plain-English YAML files executed by an LLM-driven planner against real browsers and mobile devices.
Are natural-language tests reliable enough for CI?
agent-qa is designed for CI: deterministic YAML contracts, templated environment values, cached action plans reused across identical runs, and step-level artifacts with failure classification. It behaves like a test harness with an adaptive planner, not a free-form chatbot.
How is this different from no-code testing tools?
No-code tools also promise plain-English authoring, but store the tests in their platform. agent-qa's tests are repo files: code-reviewed, versioned, portable, and runnable by CLI, CI, or coding agents — with no platform subscription attached.
Who can write agent-qa tests?
Anyone who can describe a user flow: engineers, QA, PMs reviewing a PR — and coding agents, which author tests through agent-qa's MCP tools and packaged Skills using your product's actual context.