Evaluate agent-qa on your app
Measure useful bug detection, repeated-run cost, and agent setup success with a reproducible experiment.
agent-qa is a strong fit when you want natural-language end-to-end tests, web and native mobile support, repository-owned knowledge, and a CLI/MCP workflow that a coding agent can operate. The useful test is whether those capabilities improve verification on your application.
Pick a representative journey
Choose a flow with an observable business outcome: a saved setting, a completed checkout in a test environment, or an invitation with the expected permissions. Write the expected result before running the agent. Include the normal case and a controlled regression. Use a local fixture or isolated test environment.
Keep the app revision, dataset, browser/device, test instructions, model version and execution settings in the report. Publish the fixture and commands when you can share them. Remove credentials and user data from artifacts before sharing.
Measure correctness before speed
| Measurement | What to record |
|---|---|
| Working application | How often the correct flow passes over repeated trials. |
| Controlled regression | How often the broken behavior is detected. |
| False pass | A broken flow reported as passing; inspect the assertion and evidence. |
| False failure | A correct flow reported as failing; separate runtime, test and environment causes. |
| Incomplete run | Authentication failures, cancellations and timeouts; keep these visible. |
| Agent setup | Whether the agent completes setup from the public prompt, and what human input is needed. |
Report counts and the number of trials, including unsuccessful attempts. Do not discard inconvenient runs or treat unexecuted tests as passes. Successful demonstrations show that a flow ran; they do not establish a general bug-detection rate.
Compare repeated runs
Use the same test and environment for each condition. Record total runtime, model requests, available token usage, cache hits and the outcome. Keep warm-up work separate from measured trials.
# Normal configuration, with its configured memory and cache behavior
npx agent-qa run tests/app-smoke.yaml --headless
# Compare execution with each mechanism disabled
npx agent-qa run tests/app-smoke.yaml --headless --no-cache
npx agent-qa run tests/app-smoke.yaml --headless --no-memory
npx agent-qa run tests/app-smoke.yaml --headless --no-cache --no-memoryThese flags disable use on that run; they do not prove you started with an empty knowledge store. For a cold-start experiment, use a fresh isolated workspace with its own runtime and memory paths. Keep the app and test identical. Restore the same starting state between comparisons and vary the run order to reduce warm-up bias.
The homepage's 42s → 8s cache illustration explains the intended effect of skipping repeated planning. It is not a universal benchmark. Report your measured distribution, not a promised multiplier. If exact token accounting is unavailable from a provider, say so instead of estimating it as a measured value.
Compare the right alternatives
Compare agent-qa with the workflow you would actually use: maintained Playwright or Appium tests, a coding agent driving a browser directly, or a managed QA service. Hold the behavior and acceptance criteria constant. Include test maintenance effort and agent setup effort alongside execution cost.
Existing deterministic tests remain useful. agent-qa does not replace unit tests, static checks, load testing or a dedicated security assessment. A browser-control tool may be sufficient for a one-off inspection. agent-qa adds durable test definitions, execution evidence, memory and reuse when the flow needs to be checked repeatedly.
Publish a result others can check
Include the fixture/revision, test YAML, exact commands, model and settings, trial counts, expected and observed outcomes, timing/token measurements, relevant artifacts and known limitations. The recorded demonstrations are a place to inspect agent-qa's output before running your own experiment.