---
title: "agent-qa scores 100% on iOS and 99.1% on Android in AppControlBench"
description: "agent-qa, the open-source QA agent, completed 60/60 iOS and 54.5/55 Android AppControlBench tasks in real Bluesky and Element apps. Every run is published."
canonical_url: "https://vostride.com/benchmark/app-control-bench"
md_url: "https://vostride.com/benchmark/app-control-bench.md"
last_updated: "2026-10-05T02:22:31+05:30"
---

> [Documentation index](https://vostride.com/docs.md) · [Site index](https://vostride.com/llms.txt)
> Follow the Markdown links to read individual pages; the indexes list the available documentation.

# agent-qa scores 100% on iOS and 99.1% on Android in AppControlBench

> agent-qa, the open-source QA agent, completed 60/60 iOS and 54.5/55 Android AppControlBench tasks in real Bluesky and Element apps. Every run is published.

agent-qa is an open-source self-improving QA agent. We have tested it on hundreds of web and mobile apps, and on Software Mansion's [AppControlBench](https://github.com/software-mansion/app-control-bench), where an agent operates real Bluesky and Element apps from plain-language requests, it completed all 60 iOS tasks and scored 54.5 of 55 on Android, one attempt per task.

## Comparison

| Benchmark | agent-qa (gpt-6-sol) | argent (qwen3.8-max (high)) | agent-device (qwen3.8-max (high)) | Midscene (Doubao Seed 2.1 Turbo) | No tool (qwen3.8-max (high)) |
| --- | --- | --- | --- | --- | --- |
| iOS: Completion, 60 tasks | 100.0% | 98.3% | 92.5% | 97.5% | 61.7% |
| iOS: Pass rate, 60 tasks | 100.0% | 96.7% | 88.3% | 96.7% | 50.0% |
| iOS by app: Bluesky, 30 tasks | 100.0% | 98.3% | 93.3% | 96.7% | 51.7% |
| iOS by app: Element, 30 tasks | 100.0% | 98.3% | 91.7% | 98.3% | 71.7% |
| iOS by task: Navigation, 39 tasks | 100.0% | 98.7% | 97.4% | 100.0% | 60.3% |
| iOS by task: Interaction, 13 tasks | 100.0% | 96.2% | 84.6% | 88.5% | 61.5% |
| iOS by task: Posting, 8 tasks | 100.0% | 100.0% | 81.3% | 100.0% | 68.8% |
| Android: Completion, 55 tasks | 99.1% | – | – | – | – |

Completion counts a success as 1, a partial as 0.5 and a fail as 0; pass rate counts full successes only. The argent, agent-device and No tool columns are each the best published model for that setup, from the AppControlBench leaderboard as of 2026-10-04, recomputed per app and per task from its per-task verdicts. Midscene's figures come from its own report, which rounds its completion to 98%, and agent-qa's from our runs: neither is on the leaderboard, and models, judges and iOS versions differ between the columns. The leaderboard has no Android results yet.

## The problem

When a coding agent writes the code and its tests, both carry the same assumptions. If it misread the requirement, the tests encode the mistake, and coding agents are very good at convincing themselves it works. Asking the same agent to test itself adversarially is slow, flaky and does not scale: it is the same model grading its own homework. QA needs its own agent.

## Our approach

We rebuilt QA for the agentic era. agent-qa tests on real browsers and devices and learns from every run.

- **The harness:** Battle-tested frameworks, Playwright and Appium, are the kernel that executes every action. A separate verifier must agree each step is done, and every run is recorded.
- **memidex:** A memory layer for QA: plain-text facts and procedures you review in pull requests.
- **Self-improvement loop:** After each run, agent-qa suggests what it should remember. It keeps a lesson only when runs back it up, and if you mark a run as passed or failed, your call wins over the agent's.

## AppControlBench

AppControlBench is 60 plain-language tasks in real Bluesky and Element apps. A vision judge grades the final screen: success counts 1, partial 0.5, fail 0. agent-qa completed all 60 iOS tasks and scored 54.5 of 55 on Android, one attempt per task; the benchmark's Android port, still in review, supports 55 of the tasks.

memidex and the self-improvement loop were switched off for this benchmark, and every task started in a fresh workspace with an empty memory. agent-qa ran on gpt-6-sol and the judge was gpt-6-sol (high). Each task's context added the same task-agnostic paragraph to the benchmark's preamble, on ending at a screen that shows the requested result. One iOS task, element-17, was rerun after a low-battery safeguard stopped it.

## Runs

Every run is on the page in agent-qa's run viewer, with each step, the agent's plan, its screenshots and the verifier's reasoning, next to the screen the judge graded.

### iOS

iPhone 17 Pro simulator, iOS 27.0; Element 1.11.40, Bluesky 1.122.0. Completion 100%: 60 success, 0 partial, 0 fail.

| Task | Request | Judge | Time | Actions |
| --- | --- | --- | --- | --- |
| bsky-01 | Open the "Feeds" tab (next to Following) at the top of the Bluesky app. | Success | 43s | 1 |
| bsky-02 | Open the navigation drawer / side menu in Bluesky (the menu button at the top-left). | Success | 35s | 1 |
| bsky-03 | Go to Search in Bluesky and search for the term "cat". Submit the search. | Success | 1m 8s | 2 |
| bsky-04 | In the Bluesky Following feed, open the profile of the author of the first post. | Success | 50s | 1 |
| bsky-05 | In Bluesky, open the Search / Explore tab (tap the magnifying-glass icon in the bottom navigation bar). | Success | 42s | 1 |
| bsky-06 | Open the first post in the Bluesky Following feed to view it as a thread (the post detail page with its replies). | Success | 2m 45s | 5 |
| bsky-07 | In the Bluesky Following feed, open the "..." (more options) menu on the first post - tap the three-dots icon at the end of that post's action row. | Success | 45s | 1 |
| bsky-08 | Use Bluesky Search to look up the handle "whiskers.test" and open its profile. | Success | 1m 45s | 4 |
| bsky-09 | Open a user profile from the Bluesky Following feed, then switch to that profile's "Media" tab (or "Replies" tab). | Success | 1m 12s | 2 |
| bsky-10 | Open the Bluesky Settings screen (open the left menu / drawer, then tap Settings). | Success | 45s | 2 |
| bsky-11 | In Bluesky, open your own profile by tapping the profile tab (the avatar icon at the far right of the bottom navigation bar). | Success | 42s | 1 |
| bsky-12 | Open the Notifications tab in Bluesky (the bell icon in the bottom navigation). | Success | 36s | 1 |
| bsky-13 | Open the Chat / Messages (direct messages) section of Bluesky. | Success | 40s | 1 |
| bsky-14 | In Bluesky Search, search for "test" and then switch the results to the "People" tab. | Success | 1m 56s | 5 |
| bsky-15 | In Bluesky Search, search for "news" and then switch the results to the "Feeds" tab. | Success | 1m 18s | 3 |
| bsky-16 | In Bluesky, open the hashtag feed for #caturday (search the hashtag, or tap a #caturday tag on a post, to open its dedicated feed). | Success | 5m 11s | 15 |
| bsky-17 | Open your own Bluesky profile and switch to its "Likes" sub-tab. | Success | 1m 14s | 2 |
| bsky-18 | In the Bluesky Following feed, find a post that contains an image and tap the image to open it full-screen in the image viewer. | Success | 53s | 1 |
| bsky-19 | Open Bluesky Settings and go to the Appearance screen (color mode / theme / font). | Success | 1m 24s | 3 |
| bsky-20 | Open Bluesky Settings and open the Moderation screen. | Success | 1m 6s | 3 |
| bsky-21 | Open your own Bluesky profile and tap "Edit Profile" to open the profile editor (do not change or save anything). | Success | 1m 6s | 2 |
| bsky-22 | Like the first post in the Bluesky Following feed (tap its heart icon). | Success | 41s | 1 |
| bsky-23 | Repost the first post in the Bluesky Following feed (tap the repost icon and confirm "Repost"). | Success | 1m 4s | 2 |
| bsky-24 | In Bluesky, open the profile of whiskers.test (the author of the first Following-feed post) and unfollow that account - its button currently reads "Following"; tap it to unfollow. | Success | 1m 13s | 2 |
| bsky-25 | Reply to the first post in the Bluesky Following feed with the text "nice one". | Success | 3m 46s | 9 |
| bsky-26 | In Bluesky, search for the account "newfriend.test", open its profile, and tap Follow to follow that account. | Success | 2m 35s | 7 |
| bsky-27 | In Bluesky, search for the account "loudspammer.test", open its profile, open its "..." overflow menu, and mute that account. | Success | 2m 46s | 7 |
| bsky-28 | Compose and publish a new Bluesky post with the exact text "hello from the benchmark". | Success | 1m 45s | 4 |
| bsky-29 | Compose and publish a new Bluesky post with the exact text "testing hashtags #silverbench". | Success | 1m 39s | 4 |
| bsky-30 | In the Bluesky Following feed, quote-post the first post: open its repost menu, choose "Quote post", and add the comment "sharing this". | Success | 4m 21s | 10 |
| element-01 | Open the "Team Standup" conversation in the messenger. | Success | 48s | 1 |
| element-02 | Open the direct-message (DM) conversation with Bob. | Success | 40s | 1 |
| element-03 | Open the "Weekend Plans" room and send the message "benchmark test". | Success | 1m 19s | 3 |
| element-04 | Filter the room list to find rooms matching "Book". | Success | 43s | 1 |
| element-05 | Open the app Settings (the gear icon). | Success | 59s | 2 |
| element-06 | Open the create-room menu (the + button) in the messenger. | Success | 45s | 1 |
| element-07 | Open the "Project Phoenix" room and open its room-options menu (the '...' in the header). | Success | 1m 42s | 4 |
| element-08 | Open the Team Standup room and send the message "good morning". | Success | 1m 42s | 3 |
| element-09 | Open the "Book Club" room and read its messages. | Success | 1m 3s | 2 |
| element-10 | Open Settings and find the theme / appearance preferences. | Success | 2m 47s | 5 |
| element-11 | Open the "Team Standup" room and switch to its Threads view (the threads icon in the room header). | Success | 1m 10s | 2 |
| element-12 | In the chat list, long-press the "Book Club" room and open its Notifications settings. | Success | 1m 3s | 2 |
| element-13 | Open the "Team Standup" room, view its members, and open the profile of one of its members (for example Bob). | Success | 2m 17s | 6 |
| element-14 | Open the "Team Standup" room and open its room information screen (tap the room name/header). | Success | 1m 58s | 3 |
| element-15 | Open the "Project Phoenix" room and view its list of members/people. | Success | 1m 57s | 5 |
| element-16 | Open the app Settings and go to the Notifications settings. | Success | 1m 42s | 4 |
| element-17 | Open the app Settings and go to the Security & Privacy settings. | Success | 1m 25s | 3 |
| element-18 | Open the app Settings and open the Sessions / devices list (your logged-in sessions). | Success | 1m 51s | 4 |
| element-19 | Open the app Settings and go to the Help & About section. | Success | 1m 51s | 4 |
| element-20 | Open the account/user menu by tapping your avatar/profile picture in the top-left of the chat list. | Success | 47s | 1 |
| element-21 | Open the Spaces area (the space list / space rail). | Success | 49s | 1 |
| element-22 | Open the "Team Standup" room and use its message search to search for "update". | Success | 2m 43s | 6 |
| element-23 | In the "Team Standup" room, add an emoji reaction (for example a thumbs-up) to one of the messages in the timeline. | Success | 2m 0s | 3 |
| element-24 | In the "Book Club" room, reply to one of the existing messages with the text "sounds good". | Success | 3m 5s | 6 |
| element-25 | Open the "Weekend Plans" room and open the attachment / add menu in the message composer (the + / attachment button). | Success | 1m 10s | 2 |
| element-26 | In the chat list, long-press (press and hold) the "Book Club" room to open its context menu. | Success | 55s | 1 |
| element-27 | Open the "Team Standup" room and open the screen to invite/add people to the room. | Success | 1m 4s | 2 |
| element-28 | Open the "Project Phoenix" room and send the message "status update". | Success | 1m 37s | 3 |
| element-29 | Open the direct-message conversation with Bob and send the message "hi bob". | Success | 1m 16s | 3 |
| element-30 | Open the "Book Club" room and send the message "great read". | Success | 2m 20s | 4 |

### Android

Pixel 6 emulator (AVD), Android 13 (API 33); Element X 26.08.0, Bluesky 1.122.0. Completion 99.1%: 54 success, 1 partial, 0 fail.

| Task | Request | Judge | Time | Actions |
| --- | --- | --- | --- | --- |
| bsky-01 | Open the "Feeds" tab (next to Following) at the top of the Bluesky app. | Success | 21s | 1 |
| bsky-02 | Open the navigation drawer / side menu in Bluesky (the menu button at the top-left). | Success | 21s | 1 |
| bsky-03 | Go to Search in Bluesky and search for the term "cat". Submit the search. | Success | 38s | 2 |
| bsky-04 | In the Bluesky Following feed, open the profile of the author of the first post. | Success | 1m 45s | 4 |
| bsky-05 | In Bluesky, open the Search / Explore tab (tap the magnifying-glass icon in the bottom navigation bar). | Success | 22s | 1 |
| bsky-06 | Open the first post in the Bluesky Following feed to view it as a thread (the post detail page with its replies). | Success | 24s | 1 |
| bsky-07 | In the Bluesky Following feed, open the "..." (more options) menu on the first post - tap the three-dots icon at the end of that post's action row. | Success | 20s | 1 |
| bsky-08 | Use Bluesky Search to look up the handle "whiskers.test" and open its profile. | Success | 56s | 4 |
| bsky-09 | Open a user profile from the Bluesky Following feed, then switch to that profile's "Media" tab (or "Replies" tab). | Success | 56s | 3 |
| bsky-10 | Open the Bluesky Settings screen (open the left menu / drawer, then tap Settings). | Success | 30s | 2 |
| bsky-11 | In Bluesky, open your own profile by tapping the profile tab (the avatar icon at the far right of the bottom navigation bar). | Success | 22s | 1 |
| bsky-12 | Open the Notifications tab in Bluesky (the bell icon in the bottom navigation). | Success | 26s | 1 |
| bsky-13 | Open the Chat / Messages (direct messages) section of Bluesky. | Success | 20s | 1 |
| bsky-14 | In Bluesky Search, search for "test" and then switch the results to the "People" tab. | Success | 45s | 3 |
| bsky-15 | In Bluesky Search, search for "news" and then switch the results to the "Feeds" tab. | Success | 47s | 3 |
| bsky-16 | In Bluesky, open the hashtag feed for #caturday (search the hashtag, or tap a #caturday tag on a post, to open its dedicated feed). | Success | 2m 44s | 10 |
| bsky-17 | Open your own Bluesky profile and switch to its "Likes" sub-tab. | Success | 30s | 2 |
| bsky-18 | In the Bluesky Following feed, find a post that contains an image and tap the image to open it full-screen in the image viewer. | Success | 23s | 1 |
| bsky-19 | Open Bluesky Settings and go to the Appearance screen (color mode / theme / font). | Success | 44s | 3 |
| bsky-20 | Open Bluesky Settings and open the Moderation screen. | Success | 44s | 3 |
| bsky-21 | Open your own Bluesky profile and tap "Edit Profile" to open the profile editor (do not change or save anything). | Success | 28s | 2 |
| bsky-22 | Like the first post in the Bluesky Following feed (tap its heart icon). | Success | 24s | 1 |
| bsky-23 | Repost the first post in the Bluesky Following feed (tap the repost icon and confirm "Repost"). | Success | 34s | 2 |
| bsky-24 | In Bluesky, open the profile of whiskers.test (the author of the first Following-feed post) and unfollow that account - its button currently reads "Following"; tap it to unfollow. | Partial | 15m 3s | 69 |
| bsky-25 | Reply to the first post in the Bluesky Following feed with the text "nice one". | Success | 1m 25s | 6 |
| bsky-26 | In Bluesky, search for the account "newfriend.test", open its profile, and tap Follow to follow that account. | Success | 1m 20s | 5 |
| bsky-27 | In Bluesky, search for the account "loudspammer.test", open its profile, open its "..." overflow menu, and mute that account. | Success | 1m 36s | 7 |
| bsky-28 | Compose and publish a new Bluesky post with the exact text "hello from the benchmark". | Success | 1m 15s | 5 |
| bsky-29 | Compose and publish a new Bluesky post with the exact text "testing hashtags #silverbench". | Success | 1m 13s | 5 |
| bsky-30 | In the Bluesky Following feed, quote-post the first post: open its repost menu, choose "Quote post", and add the comment "sharing this". | Success | 1m 30s | 6 |
| element-01 | Open the "Team Standup" conversation in the messenger. | Success | 18s | 1 |
| element-02 | Open the direct-message (DM) conversation with Bob. | Success | 22s | 1 |
| element-03 | Open the "Weekend Plans" room and send the message "benchmark test". | Success | 55s | 3 |
| element-04 | Filter the room list to find rooms matching "Book". | Success | 42s | 3 |
| element-05 | Open the app Settings (the gear icon). | Success | 35s | 2 |
| element-06 | Open the create-room menu (the + button) in the messenger. | Success | 17s | 1 |
| element-07 | Open the "Project Phoenix" room and open its room-options menu (the '...' in the header). | Success | 43s | 3 |
| element-08 | Open the Team Standup room and send the message "good morning". | Success | 55s | 4 |
| element-09 | Open the "Book Club" room and read its messages. | Success | 35s | 2 |
| element-10 | Open Settings and find the theme / appearance preferences. | Success | 43s | 3 |
| element-12 | In the chat list, long-press the "Book Club" room and open its Notifications settings. | Success | 1m 0s | 3 |
| element-13 | Open the "Team Standup" room, view its members, and open the profile of one of its members (for example Bob). | Success | 1m 8s | 5 |
| element-14 | Open the "Team Standup" room and open its room information screen (tap the room name/header). | Success | 54s | 3 |
| element-15 | Open the "Project Phoenix" room and view its list of members/people. | Success | 41s | 3 |
| element-16 | Open the app Settings and go to the Notifications settings. | Success | 27s | 2 |
| element-19 | Open the app Settings and go to the Help & About section. | Success | 34s | 2 |
| element-21 | Open the Spaces area (the space list / space rail). | Success | 21s | 1 |
| element-23 | In the "Team Standup" room, add an emoji reaction (for example a thumbs-up) to one of the messages in the timeline. | Success | 1m 4s | 3 |
| element-24 | In the "Book Club" room, reply to one of the existing messages with the text "sounds good". | Success | 1m 37s | 6 |
| element-25 | Open the "Weekend Plans" room and open the attachment / add menu in the message composer (the + / attachment button). | Success | 34s | 2 |
| element-26 | In the chat list, long-press (press and hold) the "Book Club" room to open its context menu. | Success | 26s | 1 |
| element-27 | Open the "Team Standup" room and open the screen to invite/add people to the room. | Success | 55s | 3 |
| element-28 | Open the "Project Phoenix" room and send the message "status update". | Success | 1m 37s | 5 |
| element-29 | Open the direct-message conversation with Bob and send the message "hi bob". | Success | 45s | 3 |
| element-30 | Open the "Book Club" room and send the message "great read". | Success | 47s | 3 |