Skip to main content

After every run, agent-qa analyzes it and learns from it. Most of the time this finishes inside the agent-qa run process and you never think about it. This page is for when it does not.

After a run

When a run has its verdict, agent-qa queues a chain of three jobs:

  1. analyze-run gives each failure a root cause.
  2. curate-memory turns the run's evidence into knowledge.
  3. propose-improvements runs extra checks on new knowledge when an eval corpus exists.

The run works through the chain for up to 60 seconds, then exits. The verdict and exit code are fixed before the chain starts, so nothing here can change them. The run prints the state of each stage on stderr, such as analysis: complete · curation: complete.

Each stage reads pending, running, complete, failed or skipped. agent-qa digest <runId> and the dashboard's run page show the same states later.

Unfinished work

Work left when the minute is up, or when you ran with --detach, stays queued on this machine:

  • The next local run finishes it. Every agent-qa run without --detach also picks up earlier runs whose curation never finished.
  • agent-qa knowledge drain finishes queued curation now, using fixed templates instead of a model. --status only reports what is left.

The queues are local files. On a CI runner or any machine that is thrown away, unfinished work goes with it.

The daemon

npx agent-qa daemon

The daemon is an optional background host that runs in the foreground of its terminal. It finishes queued analysis jobs as they arrive, and it serves agent-qa functions to commands that pass --via-daemon. It needs a working model credential to start.

CommandWhat it does
agent-qa daemon statusShows whether it is running, its uptime and how much curation is waiting. --json prints the raw status.
agent-qa daemon stopStops the running daemon.

Restart the daemon, and any long-running dashboard, after you upgrade agent-qa.

Three queues

agent-qa keeps three separate queues. Messages and commands always name which one they mean.

QueueHoldsLives inCommands
Run queueTest runs started from the dashboardThe dashboard's databaseagent-qa queue list, queue cancel
Curation queueKnowledge work from finished runsagent-qa-knowledge/memory/queue/agent-qa knowledge drain
Agent job queueThe post-run jobs above.agent-qa/agent-jobs.dbagent-qa agent-jobs …

Agent jobs

CommandWhat it does
agent-qa agent-jobs listLists every job with its kind, status, attempts and run, without payloads.
agent-qa agent-jobs pump --only <kind>Works through one kind of job now, leaving the others queued.
agent-qa agent-jobs cancel <jobId>Asks a job to stop.
agent-qa agent-jobs requeue <jobId>Returns a finished, failed or cancelled job to the queue, forgetting its previous attempt. Run pump afterwards.

Jobs retry on their own: analysis and curation up to three times, proposals twice. If analysis keeps failing, the chain carries on with an unknown root cause.

If the job store is refused

The job database is stamped with its format version, and agent-qa refuses one from another version instead of guessing. To reset it, stop agent-qa and delete .agent-qa/agent-jobs.db, its -wal and -shm files, and .agent-qa/agent-jobs/checkpoints/. Recorded runs, their evidence and your knowledge live elsewhere and are not affected.