Skip to main content
agent-qa on GitHub,230stars

> agent-qa

The self-improving Agentic QA harness with Memory

Write tests in natural language for web and mobile. agent-qa learns from past runs, adapts to UI changes, and catches regressions before you ship.

Get startedView on GitHub

15K+

test runs

1,200+

manual testing hours saved

120K+

total sub actions executed

Natural-language tests

Describe actions and assertions in natural language. agent-qa resolves them against the live interface using visible roles, labels, and screen state.

Learn about natural language tests

Natural-language YAML

Write the behavior and expected outcome in plain English. The test stays as reviewable YAML in your repository.

Targets users recognize

Refer to “New issue,” “Checkout,” or the “Issues table.” agent-qa finds the matching control in the live interface.

One format, every surface

Use the same natural-language structure across web, Android, and iOS without maintaining selector-heavy variants.

Execution memory

A run that passes leaves evidence: elements that resolved, a flow that worked, and timings that are real. agent-qa distils that evidence into durable facts, procedures, cautions and measurements. Each becomes a reviewable bundle committed next to your tests. Nothing steers a run until the evidence says it should, and once it does, the next run replays what already works instead of deriving it again.

Learn about memory

Bundles you can review

Each thing learned is a directory in your repository: the record, the evidence behind it, and the artifacts that justify it. Read it in a pull request.

Proven before it is used

New knowledge is born a candidate. Offline replay and attributed evidence from live runs decide whether it ever reaches a planner.

Warm runs get faster

A flow that already worked is replayed rather than re-derived, and stale facts are superseded instead of quietly served.

Web, Android, and iOS

Point the same natural-language flow at a web target, an Android build, or an iOS build. Playwright and Appium are execution kernels here, not code generators: no test script is ever produced. The agent decides on each action from what it can currently see, and the kernel performs it, whether that is a click in Chromium, a tap on a local emulator, or a gesture on a remote device.

Learn about mobile testing

Execution kernels

Playwright and Appium only perform the actions the agent decides on. No test script is generated, and nothing is replayed from a recording.

Three web engines

Run the same flow on Chromium, Firefox, or WebKit, and override the engine, viewport, or headless mode per test.

Native and remote devices

Drive real Android and iOS builds by app package or bundle ID, on a local emulator, a simulator, or a remote device.

Built for Humans & Agents

A test is a file somebody writes. A product manager, an engineer, and a QA engineer can each write one in natural language, and so can whatever agent your team already runs, whether that is a coding agent or a personal one, using the company context it has through agent-qa's packaged skills and MCP server. Whoever typed it, the result is the same reviewable YAML in the same repository.

Learn about MCP and skills

Anyone can author

A product manager, an engineer, and a QA engineer edit the same file, in the language they would use to describe the journey out loud.

Skills and MCP for agents

Packaged skills teach any agent to author, debug, and triage, coding or personal. The MCP server gives it the calls to validate a test, queue a run, and read the artifacts back.

One reviewable artifact

Whoever typed it, the result is the same YAML in the same repository, reviewed alongside the feature it covers.

Self-improvement

agent-qa gets better at your app on its own. After every run it works out what actually broke, learns the part worth remembering, and proves that lesson before it acts on it. Flaky tests and broken environments are set aside, so a bad afternoon never turns into a bad habit.

Learn about the curator

Diagnosis first

Every failure is diagnosed before anything is learned from it. Flaky tests and broken environments are set aside rather than studied.

Proved, then trusted

Nothing new is believed on day one. A lesson has to hold up against past runs and earn its place on a live one before the agent uses it.

It never rewrites itself

It improves what it knows about your app and suggests rule changes. It never edits its own prompts or code, and it cannot grade its own homework.

Self-healing execution

When a sub-action fails, because the click was intercepted, the control moved, or the field will not take the value, agent-qa re-observes the screen and plans another route to the same step, inside the same run. What it will not do is find another step: a recovery has to reach what you asked for, so a route to a different outcome is declined rather than taken, and the run fails with the reason it found.

Learn about self-healing

Re-observe, then re-plan

A failed sub-action is not retried. The screen is read again and the next attempt is planned against what is actually on it.

Bounded by the step

A recovery has to reach what the step asked for. A route that would reach a different outcome is declined, not taken.

The trace stays honest

Every attempt is recorded: what failed, what was tried, and what was refused. A healed step never reads as a clean one.

Caching and token optimization

agent-qa never generates a test script. Every run works out how to satisfy each step from what is on the screen at that moment, which is why a test survives a redesign that would have broken a recorded one. Working an action out costs tokens and wall clock, so a validated one is kept and reused the next time the screen still matches, and thinking is spent only on what actually changed. The suite gets faster and cheaper the longer it lives, and none of that speed comes from freezing it into a script.

Learn about caching

No script to generate

Nothing is recorded or compiled. Each run decides its own actions, so there is no generated file sitting between your app and the test.

Reused while it holds

A validated action is reused only while its target and the surrounding screen still match. Anything else goes back to the model.

Cheaper the longer it lives

Every proved run leaves less for the next one to work out, so a suite gets quicker and costs less without being frozen in place.

Execution speed

5x

42s to 8s

A run that meets screens it has already proved spends its time acting rather than deciding.

Token saving

60%

less token usage

A reused action never reaches the model, so the only thinking billed is the thinking your app made necessary.

Version controlled, built for teams

Tests, suites, hooks, and the workspace config are files you write. What it learns about your app, the bugs it files, the rules you assert, and the skills an authoring agent uses are files agent-qa writes, in a visible directory beside your tests rather than in a database you cannot read. All of it is committed, so a new memory bundle arrives as a diff in a pull request with the evidence that justified it. Only derived indexes and binary run artifacts are gitignored, because they rebuild from what is committed. A teammate, a coding agent, and CI check out one commit and get the same brief.

Learn about configuration

Files, not a database

What you author and what the agent learns sit next to each other as committed files. Nothing it knows is locked in a store you cannot open.

Learning arrives as a diff

A new memory bundle or a filed issue shows up in a pull request, with the evidence behind it, and is approved the way any other change is.

Everyone reads the same commit

A teammate, a coding agent, and CI work from one checkout, so no run is quietly using state that nobody else has.

Sandboxed hooks

Run Node, Bun, Python, or Bash scripts in isolated Docker containers: before a test as setup, at one specific step during the run, or after it finishes as teardown. Set up environments, call APIs, seed fixtures, tear down state, and pass structured output back into the active run.

Learn about hooks

Four runtimes

Write the script in Node, Bun, Python, or Bash. Each runtime has its own hook runner image.

Isolated execution

The container is read-only, capped on CPU, memory, and pids, and can be denied the network entirely.

Structured outputs

A script returns values by writing a dotenv file. Later steps read them back as variables.

Bring your own model

The model is a setting, not a rewrite. Point a workspace at an OpenAI or Anthropic compatible endpoint, at Gemini, at an open-weight model running on your own hardware, or at a subscription your team already pays for like Codex or Claude Code, and override it on the one test that needs something stronger than the rest. Nothing about how a test is written changes when the model does, because the test says what should happen and the model is only what works out how to get there. There is no vendor to be locked to and no key of ours to buy.

Learn about LLM providers

Swap it in one line

The provider, the endpoint, and the model are config. Change them and every test you have already written runs against the new one, unedited.

The seat you already pay for

Account-backed authentication works where the provider allows it, so a Codex or Claude Code subscription can drive runs without a second bill.

Or nothing leaves the building

Point it at an open-weight model on your own hardware when the code under test is the kind that cannot go to somebody else's API.

Explore live demos.

Switch between real agent-qa runs and inspect the evidence directly.

agent-qa Supabase Run
Demo for illustration only. All product names and logos are property of their respective owners.

Questions, answered.

Does agent-qa generate test code or scripts?

No. There is no generated script and no selector file. Every run reads the live screen and works out what to do from there, which sounds like repeating yourself and is not: no run is wasted, because each one feeds what it learned back in and the agent gets more accurate about your product over time. A validated action plan is cached and replayed while the screen still matches, which cuts planner tokens by around 60 percent, and the moment the interface changes under it the stale plan is purged and that step is worked out again. The suite gets faster and cheaper the longer it lives, and none of that speed comes from freezing it into a recording.

How caching works

What does it remember between runs?

What it learned about your product, written to disk as markdown under a memory root in your repository and organised by product, suite and test. It is not hidden model state. You read it in a pull request like any other file, and a new observation has to hold up against past runs before a planner is allowed to use it.

How memory works

Does the same test work on web and mobile?

Yes, which is the point of describing behaviour instead of selectors. You name the control the way you would say it out loud, the Checkout button or the issues table, and agent-qa resolves it against whatever is in front of it. Web runs use browser runtimes it installs and manages, leaving the browsers you already have alone. Android and iOS run through Appium drivers and the usual platform tooling, against an emulator, a simulator or a device on your desk.

Mobile testing guide

Which models can I use?

Any OpenAI-compatible or Anthropic-compatible endpoint, Gemini, or a local model served through something like Ollama or LM Studio. Codex and Claude Code subscriptions work too, through the optional subscription auth package. Pick a multimodal one: runs read screenshots, so text-only models are a poor fit for visual QA.

Connect a model

Is agent-qa free?

Yes. It is free to install and free to run, the source is public, and no feature is held back behind a paid tier. The tests you write and the memory it builds are ordinary files in your own repository. The only running cost is whatever your model provider charges for inference, which is the number the cache is there to keep down.

Read the license

Write your first test.

Read the quickstartView on GitHub