Write a plugin

A remote plugin connects agent-qa to a browser or device service. It handles what differs from one service to the next: creating a session before agent-qa connects and deleting it afterwards, signing the connection, building it from a few settings, uploading an app, running a tunnel, and marking and linking the session.

agent-qa handles everything else: connecting through Playwright or WebdriverIO, credentials and masking, refusals and their kinds, the heartbeat, and releasing sessions when it stops. agent-qa's own Kernel, BrowserStack, Steel and Hyperbrowser plugins are built with the same kit and pass the same conformance suite. Read one when you want a complete example.

How a plugin runsCopy Anchor Link

A plugin is an npm package, or a folder with a package.json, whose package.json carries a manifest. agent-qa starts it as a separate Node process and talks to it in newline-delimited JSON-RPC over its standard input and output. Its code never runs inside agent-qa.

A project enables it once in plugins, with the account's options, and each remote names the entry and adds its own settings:

plugins:
  - name: acme
    package: "@acme/agent-qa-plugin-acme"
    options:
      apiKey: "{{secret:ACME_API_KEY}}"

registry:
  remotes:
    - name: acme-eu
      plugin: acme
      settings: { region: eu }

Install the kitCopy Anchor Link

Add the kit as a regular dependency, not a peer or dev dependency:

npm install @vostride/agent-qa-plugin-kit

Import z from the kit, not from zod. credential() marks a field in the kit's own schema registry, and the build reads that registry, so the marks survive only when your schemas use the kit's z.

Define the pluginCopy Anchor Link

Keep the definition in a module of its own. definePlugin has no side effects, so the build step and your tests can import it without starting the plugin.

// src/plugin.ts
import {
  checkEndpoints,
  createVendorApi,
  credential,
  definePlugin,
  z,
} from '@vostride/agent-qa-plugin-kit'

function acmeApi(options: { apiKey: string; endpoint: string }) {
  return createVendorApi({
    vendor: 'Acme',
    baseUrl: options.endpoint,
    headers: { authorization: `Bearer ${options.apiKey}` },
  })
}

export default definePlugin({
  options: z.strictObject({
    apiKey: credential('The Acme API key'),
    endpoint: z.url().default('https://api.acme.example').describe('The Acme API'),
  }),
  settings: z.strictObject({
    region: z.enum(['eu', 'us']).default('eu').describe('Where the browser runs'),
  }),
  serves: ['cdp'],
  skill: './skill',

  initialize({ options }) {
    checkEndpoints(options, [{ path: 'endpoint', scheme: 'https' }])
  },

  async acquire({ options, settings }, ctx) {
    ctx.progress({ message: 'creating a browser' })
    const api = acmeApi(options)
    // The create is not cancelled: once Acme has a browser, its handle must come back to be
    // released, and the kit releases a handle that arrives after a cancel was answered.
    const response = await api.send('POST', 'browsers', {
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ region: settings.region }),
    })
    if (!response.ok) throw await api.failure(response, 'POST /browsers')
    const browser = (await api.json(response)) as { id: string; cdpUrl: string }
    return {
      handle: { id: browser.id },
      protocol: 'cdp',
      url: browser.cdpUrl,
      mask: [browser.cdpUrl],
    }
  },

  async release({ options, handle }, ctx) {
    const { id } = handle as { id: string }
    const api = acmeApi(options)
    const response = await api.send('DELETE', `browsers/${encodeURIComponent(id)}`, {
      signal: ctx.signal,
    })
    if (!response.ok && response.status !== 404) {
      throw await api.failure(response, 'DELETE /browsers/{id}', { handle })
    }
  },
})

// src/main.ts
import { runPlugin } from '@vostride/agent-qa-plugin-kit'
import plugin from './plugin.js'

runPlugin(plugin)

• options hold what the account needs and come from the plugins entry. settings hold what one place needs and come from each remote. agent-qa checks both against the schemas your plugin ships before any test starts.
• serves lists the protocols your sessions use: playwright, cdp or appium.
• acquire and release are required. initialize, check, uploadApp, started, end and shutdown are optional, and the ones you implement are the ones the manifest declares.
• Every handler takes (input, ctx). Pass ctx.signal to every request except one that creates a billed resource. ctx.progress({ message, queued?, percent? }) reports a queue position or an upload's progress, which agent-qa prints while it waits.

HandlersCopy Anchor Link

On a signal, started, end, release and shutdown get 5 s.

FailuresCopy Anchor Link

Throw PluginError with one line and a failure kind: unreachable, auth, capacity, timeout, version, refused, ended or lost.

• Add path when an option or setting is at fault, so the refusal points at the config line: options.apiKey for a field of the entry's options, settings.region for a field of the remote's settings.
• Add handle when the service had already created something, so agent-qa releases it.
• The kit checks what acquire, check, uploadApp and started return. A malformed result becomes a refused failure that names the field, and a malformed acquire result keeps its handle, so the session is still released.

Vendor API helpersCopy Anchor Link

Most plugins call one HTTP API. The kit gives every plugin the same client and the same failure lines.

createVendorApi({ vendor, baseUrl, headers?, kinds? }) builds a client for one base URL:

• send(method, path, { headers?, body?, signal? }) resolves path against baseUrl and returns the Response. headers given to the client are sent with every call and win over a call's own, so put credentials there.
• failure(response, request, { handle? }) returns a PluginError to throw, which reads <vendor> answered <status> to <request>: <reason>, with the wait on a 429 that sends Retry-After. Name the request after its route, such as DELETE /browsers/{id}, never with a real ID or key in it.
• json(response) returns the parsed body, or undefined when it is not JSON.
• A call that gets no response fails unreachable, naming only the origin.

The kind comes from the status: no response is unreachable, 401 is auth, 429 is capacity and anything else is refused. kinds adds or overrides statuses, such as { 403: 'auth', 529: 'capacity' }.

checkEndpoints(options, [{ path, scheme }]), called from initialize, refuses an endpoint option that is not https, or wss, or that carries a user or password. Plain http or ws is allowed only to this machine.

RulesCopy Anchor Link

• Standard output carries only protocol messages. Anything else there stops the plugin. The kit sends console.log, console.info, console.debug, console.warn and process.stdout.write to standard error. A child process you start, such as a tunnel, must not inherit standard output.
• Standard error is the plugin's log. agent-qa masks it and shows it only at the debug log level. Its first line becomes a failed call's reason.
• A handle is self-contained JSON. agent-qa keeps it and passes it back to started, end and release. After a crash, a fresh process with the same options releases it, so the handle carries everything release needs besides the options.
• Release is idempotent. Releasing the same handle twice, or something already gone, succeeds.
• Never cancel a create. Cancelling it loses the response that says what the service made, so nothing could release it. An acquire that fails after the service created something returns the handle in its PluginError. agent-qa never retries an acquire, and neither should a plugin.
• The kit releases on its own. When standard input closes, agent-qa's process dies or the plugin gets SIGTERM, the kit stops taking calls, aborts every running call except release and shutdown, releases every handle it still holds, runs shutdown once and exits.
• Secrets arrive only in messages. The plugin's environment holds only the path, home, temp, locale, time zone, proxy and certificate variables. Read credentials from options and settings, never from environment variables.

Run factsCopy Anchor Link

Every plugin gets the session's facts: the platform, and the browser engine or the device capabilities. Run facts reach only a plugin that declares them in the manifest's runFacts:

Declare only what you use. doctor and plugins describe show what each plugin receives.

BuildCopy Anchor Link

Build with tsdown, then emit the schemas and the manifest from the built definition:

// scripts/emit.mjs
import { emitPluginFiles } from '@vostride/agent-qa-plugin-kit'
import plugin from '../dist/plugin.js'
emitPluginFiles({ definition: plugin, packageDir: process.cwd() })

{
  "scripts": {
    "build": "tsdown && node scripts/emit.mjs"
  }
}

emitPluginFiles writes dist/options.schema.json and dist/settings.schema.json as JSON Schema draft 2020-12, with "credential": true on every marked field, and writes the manifest into package.json. Commit the manifest it writes:

{
  "name": "@acme/agent-qa-plugin-acme",
  "keywords": ["agent-qa-plugin"],
  "files": ["dist", "skill"],
  "agent-qa": {
    "plugin": {
      "kind": "remote",
      "runs": "process",
      "protocol": 1,
      "entry": "./dist/main.js",
      "serves": ["cdp"],
      "methods": [],
      "options": "./dist/options.schema.json",
      "settings": "./dist/settings.schema.json",
      "skill": "./skill"
    }
  }
}

agent-qa reads the manifest without running the plugin, and every path in it must resolve inside the package. Add the agent-qa-plugin keyword so people can find the package on npm.

The skillCopy Anchor Link

A plugin may ship one skill for coding agents: plain files, no code, in the folder the manifest's skill names. Put a SKILL.md with a name and a description in its front matter, and list the folder in the package's files. Say what each option and setting does and how to fix each refusal, the way the first-party plugins' skills do.

TestCopy Anchor Link

The kit's testing entry runs a plugin the way agent-qa does. Build before you test, because it starts the built entry:

import { fileURLToPath } from 'node:url'
import {
  expectManifestCurrent,
  runConformance,
  startLoopbackFake,
} from '@vostride/agent-qa-plugin-kit/testing'
import { expect, it } from 'vitest'
import plugin from '../plugin.js'

const packageDir = fileURLToPath(new URL('../..', import.meta.url))

it('keeps the committed manifest current', () => {
  expectManifestCurrent(packageDir, plugin)
})

it('passes conformance against a loopback fake of the Acme API', async () => {
  const api = await startLoopbackFake({
    routes: {
      'POST /browsers': [{ status: 201, json: { id: 'b-1', cdpUrl: 'ws://127.0.0.1:9/b-1' } }],
      'DELETE /browsers/:id': [{ status: 204 }],
    },
  })
  try {
    const report = await runConformance({
      packageDir,
      options: { apiKey: 'test-key', endpoint: api.url },
      settings: { region: 'eu' },
    })
    expect(report.checks.filter((check) => !check.ok)).toEqual([])
  } finally {
    await api.close()
  }
})

runConformance starts the plugin with the environment agent-qa gives it and reports each check:

quick: true runs only the checks that need no network and never calls acquire. startLoopbackFake({ routes }) serves a scripted fake of the service's API on 127.0.0.1: a route is 'METHOD /path', with :name matching one segment, and serves its replies in order. expectManifestCurrent fails when the committed manifest differs from the one the build would write.

Develop with a path entryCopy Anchor Link

While you develop, point an entry at the plugin's folder with path instead of package:

plugins:
  - name: acme
    path: ./tools/acme-plugin
    options:
      apiKey: "{{secret:ACME_API_KEY}}"

The folder must be inside the config's folder, after links are resolved, and its entry built and ready to run. A plugin elsewhere in a monorepo is a workspace package; name it with package.

Check before you publishCopy Anchor Link

npx agent-qa plugin validate ./tools/acme-plugin

plugin validate takes a folder or an installed package name, and checks the plugin the way agent-qa will load it: the manifest, both schemas, the credential marks, the skill, the package, and a quick conformance pass that never calls acquire:

ok    manifest
ok    schemas
ok    credential-marks
ok    skill
ok    package
ok    handshake  initialize refused: answered -32000: options.apiKey: Invalid input: expected string, received undefined (kind refused)
ok    unknown-method
ok    clean-stdout
ok    exit-on-stdin-close

The quick pass starts the plugin without options, so a plugin that refuses initialize for a missing option still passes handshake. credential-marks warns about a string field whose name looks like a credential but carries no credential() mark. package refuses install scripts and peerDependencies on agent-qa packages.

Compatibility and namingCopy Anchor Link

• A plugin must speak protocol N or N-1, where N is the newest protocol agent-qa speaks. Once N+1 ships, an N-1 plugin is refused with a line naming the kit version to rebuild with. Rebuilding with a newer kit is the whole upgrade.
• Do not declare peerDependencies on agent-qa or the kit. The protocol number, not a version range, decides what works together.
• Name the package agent-qa-plugin-<id> or @scope/agent-qa-plugin-<id>. This is a convention: agent-qa takes package exactly as written.
• A plugin runs on Node 24 or later, with agent-qa's own Node binary.