Skip to main content
agent-qa on GitHub,904stars
agent-qa

v0.1.x to v0.2.x

Copy page
View as MarkdownOpen the raw Markdown source.Open in ChatGPTAsk questions about this page.Open in ClaudeAsk questions about this page.Open in MistralAsk questions about this page.Open in VS CodeAsk Copilot Chat about this page.Open in CursorAsk Cursor about this page.

What is new in agent-qa v0.2, and how to move a v0.1 project across in a few minutes.

agent-qa v0.2 is here

We are excited to announce agent-qa v0.2. It took a while, and we think it was worth the wait.

After talking to the people who use agent-qa, the same problems kept coming up:

  • Memory got in the way. v0.1 kept loose notes per product, suite and test. Nothing was ever retired, so stale notes crowded the model's context.
  • Lessons were trusted too early. Something seen once counted as much as something seen fifty times.
  • Feedback had nowhere to go. When a verdict was wrong, you could fix the test, but agent-qa learned nothing from it.
  • Coding agents had to guess. They could start a run, but reading the result meant digging through files.

v0.2 is a rebuild around those problems.

What's new

  • memidex, a new memory. Each lesson is a small record with its evidence, and its trust is worked out from what later runs saw. Records are plain files in agent-qa-knowledge/ that you can review, diff and revert. Read about memidex.
  • Self-improvement you can follow. After every run agent-qa analyzes failures, proposes lessons and checks them before they are used. Local runs apply them; CI runs only propose. How it works.
  • Feedback that sticks. Mark a verdict as wrong with agent-qa runs mark and the correction applies at once. Mark a run.
  • Policies. Your QA rules move to agent-qa-knowledge/policies/ and reach the agent word for word on every run. Policies.
  • Built for coding agents. 20 MCP tools, four skills and --json output on every read command let Claude Code, Codex or any MCP client write a test, run it and read the evidence. Coding agents.
  • Ask and chat. agent-qa ask and agent-qa chat put every capability behind one conversation. Ask and chat.
  • Flaky tests handled. A test that keeps failing for flaky or environmental reasons is quarantined, so it stops failing your run. Quarantine.
  • Setup without menus. agent-qa init --llm <provider> --model <id> configures a project in one command, and the examples run against a playground shop out of the box.

How to upgrade

v0.2 does not upgrade a v0.1 project in place. Delete agent-qa's local state and initialize again. Your tests, .env and .env.secrets.local stay as they are, and the whole thing takes a few minutes.

The fastest way is to hand it to your coding agent:

Upgrade this project to agent-qa v0.2

Upgrade this project from agent-qa v0.1 to v0.2. Do the work, rather than only describing it.

Read the upgrade guide first: https://vostride.com/docs/agent-qa/migrate/v0-1-to-v0-2.md

v0.2 does not upgrade a v0.1 project in place. Re-initialize it and carry over what I wrote:

1. Find agent-qa.config.yaml and the files it names. Stop any agent-qa dashboard or MCP server you started.
2. Copy the rules file named by workspace.agentRules (usually agent-rules.md), agent-qa.config.yaml, hooks.yaml and agent-qa.local.yaml into a before-v0.2/ folder. If agent-qa-knowledge/policies/ exists from a v0.2 preview, copy its .md files too.
3. Delete .agent-qa/, agent-qa-memory/ and agent-qa-knowledge/, and the v0.1 example files the guide lists, unless one was turned into a test of my own.
4. Upgrade agent-qa with this repository's package manager. If the project uses a Claude or Codex subscription, upgrade @vostride/agent-qa-subscription-auth too.
5. Run npx agent-qa init --force with the --llm value that matches the old model: claude-code, codex, openai-compatible, anthropic-compatible or gemini. Add --model and --base-url when the old config set them.
6. Move the old rules into agent-qa-knowledge/policies/global.md, one rule per line below its comment block. Add my targets, devices and settings back to the new agent-qa.config.yaml, using the key changes in the guide. Remove step maxAttempts, use.healing and use.planner.maxSubActions from my tests and suites.
7. Run npx agent-qa validate and fix what it reports.
8. For a Claude subscription, ask me to run npx agent-qa auth login --config claude-code and finish the browser sign-in; for Codex, use --config codex. API keys saved for the default entry keep working.
9. Run one of my own tests or suites and report its verdict.

Never print secrets or put credentials in YAML or commands. Report what you changed, what could not be carried over, and anything I still need to do.

To do it by hand, run these steps from the folder that holds agent-qa.config.yaml.

  1. Stop agent-qa

    Stop the dashboard with Ctrl+C, and close any coding agent session that started agent-qa mcp.

  2. Set aside what you wrote

    Keep a copy of your rules and your settings. You will bring them back after the reset.

    mkdir before-v0.2
cp agent-rules.md agent-qa.config.yaml hooks.yaml agent-qa.local.yaml before-v0.2/

    If your config names a different rules file in workspace.agentRules, copy that one instead of agent-rules.md. If you tried a v0.2 preview, also copy agent-qa-knowledge/policies/*.md.

  3. Delete the old state

    rm -rf .agent-qa/ agent-qa-memory/ agent-qa-knowledge/

    Then delete the examples that v0.1 generated, unless you turned one into a test of your own:

    rm -f tests/example-pass.yaml tests/example-fail.yaml tests/hacker-news-top-story.yaml tests/bad-a11y.yaml
rm -rf tests/automation-exercise/
rm -f suites/automation-exercise.suite.yaml scripts/fetch-hn-top-story.mjs agent-rules.md

  4. Upgrade the packages

    npm install -D agent-qa@latest @vostride/agent-qa-subscription-auth@latest

    If you use an API key and never installed the subscription plugin, npm install -D agent-qa@latest is enough.

  5. Initialize again

    npx agent-qa init --force --llm claude-code

    Pick the --llm value for your model: claude-code, codex, openai-compatible, anthropic-compatible or gemini. --force rewrites the config, hooks.yaml, agent-qa.local.yaml and the examples. It never touches your other tests, .env or .env.secrets.local.

  6. Bring your settings back

    • Paste your rules into agent-qa-knowledge/policies/global.md, below its comment block, one rule per line.
    • Add your targets, devices and model settings back to agent-qa.config.yaml. A few keys changed; the table below lists them.
    • Add KOMAZON_PASSWORD=komazon123 to .env.secrets.local so the new examples can sign in. It is the public password of the playground shop the examples test.

  7. Sign in and run

    v0.1 named the Claude subscription entry claude-subscription. v0.2 names it claude-code, so sign in once more:

    npx agent-qa auth login --config claude-code

    Use --config codex for a Codex subscription. API keys saved for the default entry keep working. Then check the project and run the examples:

    npx agent-qa validate
npx agent-qa run suites/examples.suite.yaml

    Expect one pass and one failure, which is the playground's deliberate bug, and exit code 1.

What carries over

Carries over
Your tests and suitesYes. Remove the three v0.1-only keys listed below if you used them.
.env and .env.secrets.localYes. init never writes over them.
API keys in ~/.agent-qa/auth.jsonYes.
Claude subscription sign-inNo. Sign in again with --config claude-code.
Run history, dashboard data and the action cacheNo.
Saved logins from agent-qa auth-stateNo. Capture them again.
v0.1 memory in agent-qa-memory/No. memidex learns again from your next runs.

What changed in your files

Config

v0.1v0.2
workspace.agentRulesagent-qa-knowledge/policies/global.md
services.memoryknowledge
use.planner.maxSubActionsuse.planner.maxActions and use.planner.maxPlanningTurns, 30 each by default
registry.llms[].screenshotSize and effectiveResolutionservices.vision.screenshotSize and services.vision.effectiveResolution
use.healingRemoved

Tests and suites. Remove step maxAttempts, use.healing and use.planner.maxSubActions. Everything else in a v0.1 test is valid in v0.2.

CLI

v0.1v0.2
agent-qa ids generate, create-test, create-suiteRemoved. New tests get their IDs from the dashboard's test editor or from agent_qa_write_test.
agent-qa clean-memoryRemoved
run --no-cacherun --disable-cache
run --no-memoryRemoved. Memory is always on.
init --skip-installRemoved

MCP

v0.1 toolv0.2 tool
agent_qa_create_testagent_qa_write_test
agent_qa_update_testagent_qa_refine_test
agent_qa_read_testagent_qa_get_entity
agent_qa_create_hookagent_qa_change_config
agent_qa_enqueue_test_run, agent_qa_enqueue_suite_runagent_qa_start_run

The other v0.1 tools are gone. Calling any v0.1 name returns an error that names its replacement or what to do instead. The MCP reference lists every v0.2 tool.

If you skip the upgrade

Every v0.2 command that loads a v0.1 config stops and prints the steps above:

This config was written for agent-qa before v2. v2 does not upgrade a workspace; re-create it:

agent-qa doctor also warns when an old agent-qa-memory/ folder is still around. It is safe to delete.

Plugins

Add sign-in flows with auth plugins, and new model providers with provider plugins.

On this page

agent-qa v0.2 is hereWhat's newHow to upgradeStop agent-qaSet aside what you wroteDelete the old stateUpgrade the packagesInitialize againBring your settings backSign in and runWhat carries overWhat changed in your filesIf you skip the upgrade