Skip to main content
Skip to Content

agent-qa drives your real app with a model in the loop, so it is worth knowing exactly what it protects and what it leaves to you.

Secrets

Values in your secrets file, workspace.secretsFile (.env.secrets.local by default), are masked as [secret:<name>] everywhere text leaves agent-qa: model requests, knowledge records, logs, the terminal, the dashboard, MCP results, JUnit reports and analytics. A masked value is never restored.

  • Type a secret in a step with {{secret:NAME}}. The value is filled in at the moment of the action.
  • An action that contains the literal text [secret:NAME] is refused, because it would type the placeholder.
  • Steps that type a secret are never cached.
  • A secret used by a remote, a plugin entry's options, a device's capabilities or a target's app.ref is used only for that connection. A step or a target URL that names it is refused before the run starts, and hooks never receive it.

agent-qa also masks the forms it builds from a secret, such as its percent-encoded form or a remote's Basic credentials, and every value a plugin marks as a credential, shown as [secret:PLUGIN_<ENTRY>]. Nothing else is masked. Values in .env, other environment variables, cookies and saved logins, email addresses and other personal data, screenshots and session videos are kept as they are. Use temporary test accounts with limited access, and put anything sensitive in the secrets file.

Target environments

A target can say which environment it points at:

registry:
  targets:
    checkout-prod:
      platform: web
      url: https://shop.example.com
      environment: production

environment is local, shared or production, and it is a label only. agent-qa runs every target the same way and refuses none, so point tests at production only when you mean to, and use test accounts there. A config that still sets allowProduction is refused with allowProduction was removed: production targets run like any other target. Delete this line.

No confirmations

agent-qa has no approval prompts. Every command, MCP tool and chat request applies in the call: runs start, tests are written, knowledge changes. Give an agent the agent-qa MCP server the way you would give it a shell in your project.

Local only

The dashboard, its run service and agent-qa mcp --http listen on 127.0.0.1 only. The dashboard refuses to start on any other address, and requests from other sites or hosts are rejected. None of them uses tokens, so reach them from another machine only through a tunnel or proxy that authenticates for you.

Hooks run in a sandbox

Hooks run in Docker containers with 512 MB of memory, one CPU and a read-only file system. They have network access unless the hook sets network: false. A hook receives the run's variables and the declared secrets as environment variables, except secrets that only connect to a remote, plugin or device, so review hook scripts like any code that handles credentials.

App content is data

Text from your app reaches the model as observations, wrapped as untrusted data. Learned knowledge is served as hints to weigh against the screen, never as instructions. agent-qa does not try to detect prompt injection, so test hostile content in an isolated environment.

Plugins

Plugins connect agent-qa to hosted browsers and device clouds. Only @vostride/agent-qa-* packages load inside agent-qa. Any other plugin runs as its own process, started with a short list of environment variables, and receives credentials only in its own messages. agent-qa checks its options and settings against the plugin's schemas before a test starts, and masks what the plugin marks as a credential. A plugin still runs with your user's access to files and the network, so install plugins you trust.

The services themselves keep their own records. Some, such as Steel and Hyperbrowser, record sessions and keep the replay on their dashboard, where agent-qa's masking does not reach.

Usage analytics

Released versions of agent-qa send anonymous usage events to PostHog, at us.i.posthog.com, from run, mcp, chat, the daemon and init. The dashboard, ask, fn, runs, digest and entities send nothing.

Events hold only fixed values, numbers and IDs: the command, the platform, the verdict, durations, step and token counts, the agent-qa version, which coding agent ran it, whether a test ran here or on a remote, and which of vostride's plugins served it. Any other plugin is recorded as third_party. They never contain step text, page content or file paths, and they pass through the same secret masking. Your anonymous ID lives in ~/.agent-qa/analytics.json; CI runs use a fresh ID each time. PostHog may estimate a coarse location from your IP address.

To opt out, add this to agent-qa.config.yaml:

analytics:
  privacy: true

Update check

At most once a day, agent-qa run and the dashboard ask the npm registry whether a newer agent-qa exists. --quiet silences it.

What to keep out of git and CI uploads

  • .agent-qa/, except run folders you upload on purpose. Its runs.db holds config snapshots that include your .env values.
  • Saved logins in .agent-qa/auth-states/.
  • .env and the secrets file.
  • The knowledge folder's lock files and queue, which its own .gitignore already excludes.