---
title: "agent-qa is #1 on AndroidWorld benchmark"
description: "agent-qa, the open-source self-improving QA agent, passed all 116 tasks of Google's AndroidWorld benchmark. Every run is published with steps, screenshots and video."
canonical_url: "https://vostride.com/benchmark/android-world"
md_url: "https://vostride.com/benchmark/android-world.md"
last_updated: "2026-09-29T02:32:55+05:30"
---

> [Documentation index](https://vostride.com/docs.md) · [Site index](https://vostride.com/llms.txt)
> Follow the Markdown links to read individual pages; the indexes list the available documentation.

# agent-qa is #1 on AndroidWorld benchmark

> agent-qa, the open-source self-improving QA agent, passed all 116 tasks of Google's AndroidWorld benchmark. Every run is published with steps, screenshots and video.

agent-qa is an open-source self-improving QA agent. We have tested it on hundreds of web and mobile apps, and on Google's [AndroidWorld benchmark](https://github.com/google-research/android_world) it passed 116 of 116 tasks, beating Google's Artemis (99.1%).

## The problem

When a coding agent writes the code and its tests, both carry the same assumptions. If it misread the requirement, the tests encode the mistake, and coding agents are very good at convincing themselves it works. Asking the same agent to test itself adversarially is slow, flaky and does not scale: it is the same model grading its own homework. QA needs its own agent.

## Our approach

We rebuilt QA for the agentic era. agent-qa tests on real browsers and devices and learns from every run.

- **The harness:** Battle-tested frameworks, Playwright and Appium, are the kernel that executes every action. A separate verifier must agree each step is done, and every run is recorded.
- **memidex:** A memory layer for QA: plain-text facts and procedures you review in pull requests.
- **Self-improvement loop:** After each run, agent-qa suggests what it should remember. It keeps a lesson only when runs back it up, and if you mark a run as passed or failed, your call wins over the agent's.

## AndroidWorld

AndroidWorld is 116 tasks across 20 real Android apps, scored by its own evaluators. agent-qa passed 116 of 116. Every run is published below with its steps, screenshots and video.

memidex and the self-improvement loop were switched off for this benchmark. Every task started in a fresh workspace with an empty memory, and nothing learned in one run carried over to the next.

- **Model:** gpt-6-sol
- **AndroidWorld commit:** `3e50888527ef9f29b9157ecd537e408008bb1c85`
- **Leaderboard source:** the AndroidWorld community leaderboard, as of 2026-09-26

## Leaderboard

| Agent | Model | Success rate |
| --- | --- | --- |
| agent-qa | gpt-6-sol | 100.0% |
| Artemis | Gemini 3.7 Flash, Gemini Robotics ER 2 | 99.1% |
| AGI-0 | AGI-0 | 97.4% |
| Midscene.js | Gemini-3.5-Flash | 93.1% |
| DroidRun | GPT5, Gemini 2.5 Pro | 91.4% |
| Human | AndroidWorld paper baseline | 80.0% |

## Runs

Each task links to its run: the goal, every action with the agent's reasoning, and the verdict.

| Task | App | Result | Time | Actions |
| --- | --- | --- | --- | --- |
| [Audio Recorder Record Audio](https://vostride.com/benchmark/android-world/AudioRecorderRecordAudio.md) | Audio Recorder | Passed | 1m 2s | 6 |
| [Audio Recorder Record Audio With File Name](https://vostride.com/benchmark/android-world/AudioRecorderRecordAudioWithFileName.md) | Audio Recorder | Passed | 1m 46s | 7 |
| [Recipe Add Multiple Recipes](https://vostride.com/benchmark/android-world/RecipeAddMultipleRecipes.md) | Broccoli | Passed | 7m 59s | 46 |
| [Recipe Add Multiple Recipes From Image](https://vostride.com/benchmark/android-world/RecipeAddMultipleRecipesFromImage.md) | Broccoli | Passed | 8m 25s | 47 |
| [Recipe Add Multiple Recipes From Markor](https://vostride.com/benchmark/android-world/RecipeAddMultipleRecipesFromMarkor.md) | Broccoli | Passed | 14m 58s | 66 |
| [Recipe Add Multiple Recipes From Markor2](https://vostride.com/benchmark/android-world/RecipeAddMultipleRecipesFromMarkor2.md) | Broccoli | Passed | 9m 26s | 50 |
| [Recipe Add Single Recipe](https://vostride.com/benchmark/android-world/RecipeAddSingleRecipe.md) | Broccoli | Passed | 2m 18s | 13 |
| [Recipe Delete Duplicate Recipes](https://vostride.com/benchmark/android-world/RecipeDeleteDuplicateRecipes.md) | Broccoli | Passed | 1m 55s | 10 |
| [Recipe Delete Duplicate Recipes2](https://vostride.com/benchmark/android-world/RecipeDeleteDuplicateRecipes2.md) | Broccoli | Passed | 4m 55s | 25 |
| [Recipe Delete Duplicate Recipes3](https://vostride.com/benchmark/android-world/RecipeDeleteDuplicateRecipes3.md) | Broccoli | Passed | 6m 14s | 30 |
| [Recipe Delete Multiple Recipes](https://vostride.com/benchmark/android-world/RecipeDeleteMultipleRecipes.md) | Broccoli | Passed | 2m 20s | 14 |
| [Recipe Delete Multiple Recipes With Constraint](https://vostride.com/benchmark/android-world/RecipeDeleteMultipleRecipesWithConstraint.md) | Broccoli | Passed | 11m 12s | 67 |
| [Recipe Delete Multiple Recipes With Noise](https://vostride.com/benchmark/android-world/RecipeDeleteMultipleRecipesWithNoise.md) | Broccoli | Passed | 3m 15s | 18 |
| [Recipe Delete Single Recipe](https://vostride.com/benchmark/android-world/RecipeDeleteSingleRecipe.md) | Broccoli | Passed | 1m 17s | 6 |
| [Recipe Delete Single With Recipe With Noise](https://vostride.com/benchmark/android-world/RecipeDeleteSingleWithRecipeWithNoise.md) | Broccoli | Passed | 1m 32s | 8 |
| [Camera Take Photo](https://vostride.com/benchmark/android-world/CameraTakePhoto.md) | Camera | Passed | 1m 12s | 4 |
| [Camera Take Video](https://vostride.com/benchmark/android-world/CameraTakeVideo.md) | Camera | Passed | 1m 32s | 9 |
| [Browser Draw](https://vostride.com/benchmark/android-world/BrowserDraw.md) | Chrome | Passed | 2m 44s | 16 |
| [Browser Maze](https://vostride.com/benchmark/android-world/BrowserMaze.md) | Chrome | Passed | 3m 2s | 17 |
| [Browser Multiply](https://vostride.com/benchmark/android-world/BrowserMultiply.md) | Chrome | Passed | 3m 28s | 19 |
| [Clock Stop Watch Paused Verify](https://vostride.com/benchmark/android-world/ClockStopWatchPausedVerify.md) | Clock | Passed | 1m 37s | 4 |
| [Clock Stop Watch Running](https://vostride.com/benchmark/android-world/ClockStopWatchRunning.md) | Clock | Passed | 38s | 2 |
| [Clock Timer Entry](https://vostride.com/benchmark/android-world/ClockTimerEntry.md) | Clock | Passed | 57s | 6 |
| [Contacts Add Contact](https://vostride.com/benchmark/android-world/ContactsAddContact.md) | Contacts | Passed | 1m 21s | 7 |
| [Contacts New Contact Draft](https://vostride.com/benchmark/android-world/ContactsNewContactDraft.md) | Contacts | Passed | 2m 59s | 16 |
| [Files Delete File](https://vostride.com/benchmark/android-world/FilesDeleteFile.md) | Files | Passed | 2m 47s | 15 |
| [Files Move File](https://vostride.com/benchmark/android-world/FilesMoveFile.md) | Files | Passed | 2m 33s | 13 |
| [Notes Is Todo](https://vostride.com/benchmark/android-world/NotesIsTodo.md) | Joplin | Passed | 25s | 2 |
| [Notes Meeting Attendee Count](https://vostride.com/benchmark/android-world/NotesMeetingAttendeeCount.md) | Joplin | Passed | 1m 16s | 6 |
| [Notes Recipe Ingredient Count](https://vostride.com/benchmark/android-world/NotesRecipeIngredientCount.md) | Joplin | Passed | 59s | 5 |
| [Notes Todo Item Count](https://vostride.com/benchmark/android-world/NotesTodoItemCount.md) | Joplin | Passed | 2m 34s | 13 |
| [Markor Add Note Header](https://vostride.com/benchmark/android-world/MarkorAddNoteHeader.md) | Markor | Passed | 2m 52s | 16 |
| [Markor Change Note Content](https://vostride.com/benchmark/android-world/MarkorChangeNoteContent.md) | Markor | Passed | 2m 3s | 13 |
| [Markor Create Folder](https://vostride.com/benchmark/android-world/MarkorCreateFolder.md) | Markor | Passed | 51s | 4 |
| [Markor Create Note](https://vostride.com/benchmark/android-world/MarkorCreateNote.md) | Markor | Passed | 1m 4s | 6 |
| [Markor Create Note And Sms](https://vostride.com/benchmark/android-world/MarkorCreateNoteAndSms.md) | Markor | Passed | 2m 21s | 15 |
| [Markor Create Note From Clipboard](https://vostride.com/benchmark/android-world/MarkorCreateNoteFromClipboard.md) | Markor | Passed | 1m 25s | 8 |
| [Markor Delete All Notes](https://vostride.com/benchmark/android-world/MarkorDeleteAllNotes.md) | Markor | Passed | 1m 17s | 7 |
| [Markor Delete Newest Note](https://vostride.com/benchmark/android-world/MarkorDeleteNewestNote.md) | Markor | Passed | 47s | 4 |
| [Markor Delete Note](https://vostride.com/benchmark/android-world/MarkorDeleteNote.md) | Markor | Passed | 53s | 4 |
| [Markor Edit Note](https://vostride.com/benchmark/android-world/MarkorEditNote.md) | Markor | Passed | 46s | 4 |
| [Markor Merge Notes](https://vostride.com/benchmark/android-world/MarkorMergeNotes.md) | Markor | Passed | 3m 41s | 17 |
| [Markor Move Note](https://vostride.com/benchmark/android-world/MarkorMoveNote.md) | Markor | Passed | 2m 1s | 12 |
| [Markor Transcribe Receipt](https://vostride.com/benchmark/android-world/MarkorTranscribeReceipt.md) | Markor | Passed | 2m 4s | 9 |
| [Markor Transcribe Video](https://vostride.com/benchmark/android-world/MarkorTranscribeVideo.md) | Markor | Passed | 8m 30s | 41 |
| [Sports Tracker Activities Count For Week](https://vostride.com/benchmark/android-world/SportsTrackerActivitiesCountForWeek.md) | OpenTracks | Passed | 1m 48s | 10 |
| [Sports Tracker Activities On Date](https://vostride.com/benchmark/android-world/SportsTrackerActivitiesOnDate.md) | OpenTracks | Passed | 7m 58s | 32 |
| [Sports Tracker Activity Duration](https://vostride.com/benchmark/android-world/SportsTrackerActivityDuration.md) | OpenTracks | Passed | 1m 19s | 5 |
| [Sports Tracker Longest Distance Activity](https://vostride.com/benchmark/android-world/SportsTrackerLongestDistanceActivity.md) | OpenTracks | Passed | 1m 36s | 8 |
| [Sports Tracker Total Distance For Category Over Interval](https://vostride.com/benchmark/android-world/SportsTrackerTotalDistanceForCategoryOverInterval.md) | OpenTracks | Passed | 2m 0s | 11 |
| [Sports Tracker Total Duration For Category This Week](https://vostride.com/benchmark/android-world/SportsTrackerTotalDurationForCategoryThisWeek.md) | OpenTracks | Passed | 1m 15s | 6 |
| [Osm And Favorite](https://vostride.com/benchmark/android-world/OsmAndFavorite.md) | OsmAnd | Passed | 1m 33s | 7 |
| [Osm And Marker](https://vostride.com/benchmark/android-world/OsmAndMarker.md) | OsmAnd | Passed | 2m 15s | 12 |
| [Osm And Track](https://vostride.com/benchmark/android-world/OsmAndTrack.md) | OsmAnd | Passed | 11m 30s | 58 |
| [Open App Task Eval](https://vostride.com/benchmark/android-world/OpenAppTaskEval.md) | Other | Passed | 1m 33s | 8 |
| [Expense Add Multiple](https://vostride.com/benchmark/android-world/ExpenseAddMultiple.md) | Pro Expense | Passed | 10m 58s | 45 |
| [Expense Add Multiple From Gallery](https://vostride.com/benchmark/android-world/ExpenseAddMultipleFromGallery.md) | Pro Expense | Passed | 6m 19s | 31 |
| [Expense Add Multiple From Markor](https://vostride.com/benchmark/android-world/ExpenseAddMultipleFromMarkor.md) | Pro Expense | Passed | 5m 51s | 27 |
| [Expense Add Single](https://vostride.com/benchmark/android-world/ExpenseAddSingle.md) | Pro Expense | Passed | 2m 51s | 12 |
| [Expense Delete Duplicates](https://vostride.com/benchmark/android-world/ExpenseDeleteDuplicates.md) | Pro Expense | Passed | 2m 57s | 13 |
| [Expense Delete Duplicates2](https://vostride.com/benchmark/android-world/ExpenseDeleteDuplicates2.md) | Pro Expense | Passed | 2m 34s | 13 |
| [Expense Delete Multiple](https://vostride.com/benchmark/android-world/ExpenseDeleteMultiple.md) | Pro Expense | Passed | 2m 34s | 15 |
| [Expense Delete Multiple2](https://vostride.com/benchmark/android-world/ExpenseDeleteMultiple2.md) | Pro Expense | Passed | 2m 58s | 18 |
| [Expense Delete Single](https://vostride.com/benchmark/android-world/ExpenseDeleteSingle.md) | Pro Expense | Passed | 2m 12s | 8 |
| [Retro Create Playlist](https://vostride.com/benchmark/android-world/RetroCreatePlaylist.md) | Retro Music | Passed | 3m 16s | 19 |
| [Retro Playing Queue](https://vostride.com/benchmark/android-world/RetroPlayingQueue.md) | Retro Music | Passed | 17m 38s | 21 |
| [Retro Playlist Duration](https://vostride.com/benchmark/android-world/RetroPlaylistDuration.md) | Retro Music | Passed | 7m 47s | 44 |
| [Retro Save Playlist](https://vostride.com/benchmark/android-world/RetroSavePlaylist.md) | Retro Music | Passed | 3m 59s | 22 |
| [System Bluetooth Turn Off](https://vostride.com/benchmark/android-world/SystemBluetoothTurnOff.md) | Settings | Passed | 43s | 3 |
| [System Bluetooth Turn Off Verify](https://vostride.com/benchmark/android-world/SystemBluetoothTurnOffVerify.md) | Settings | Passed | 38s | 3 |
| [System Bluetooth Turn On](https://vostride.com/benchmark/android-world/SystemBluetoothTurnOn.md) | Settings | Passed | 43s | 3 |
| [System Bluetooth Turn On Verify](https://vostride.com/benchmark/android-world/SystemBluetoothTurnOnVerify.md) | Settings | Passed | 40s | 3 |
| [System Brightness Max](https://vostride.com/benchmark/android-world/SystemBrightnessMax.md) | Settings | Passed | 1m 17s | 5 |
| [System Brightness Max Verify](https://vostride.com/benchmark/android-world/SystemBrightnessMaxVerify.md) | Settings | Passed | 1m 20s | 5 |
| [System Brightness Min](https://vostride.com/benchmark/android-world/SystemBrightnessMin.md) | Settings | Passed | 1m 3s | 5 |
| [System Brightness Min Verify](https://vostride.com/benchmark/android-world/SystemBrightnessMinVerify.md) | Settings | Passed | 1m 6s | 5 |
| [System Copy To Clipboard](https://vostride.com/benchmark/android-world/SystemCopyToClipboard.md) | Settings | Passed | 2m 13s | 11 |
| [System Wifi Turn Off](https://vostride.com/benchmark/android-world/SystemWifiTurnOff.md) | Settings | Passed | 46s | 4 |
| [System Wifi Turn Off Verify](https://vostride.com/benchmark/android-world/SystemWifiTurnOffVerify.md) | Settings | Passed | 50s | 4 |
| [System Wifi Turn On](https://vostride.com/benchmark/android-world/SystemWifiTurnOn.md) | Settings | Passed | 1m 0s | 5 |
| [System Wifi Turn On Verify](https://vostride.com/benchmark/android-world/SystemWifiTurnOnVerify.md) | Settings | Passed | 1m 14s | 4 |
| [Turn Off Wifi And Turn On Bluetooth](https://vostride.com/benchmark/android-world/TurnOffWifiAndTurnOnBluetooth.md) | Settings | Passed | 1m 16s | 6 |
| [Turn On Wifi And Open App](https://vostride.com/benchmark/android-world/TurnOnWifiAndOpenApp.md) | Settings | Passed | 1m 50s | 9 |
| [Simple Calendar Add One Event](https://vostride.com/benchmark/android-world/SimpleCalendarAddOneEvent.md) | Simple Calendar | Passed | 3m 4s | 19 |
| [Simple Calendar Add One Event In Two Weeks](https://vostride.com/benchmark/android-world/SimpleCalendarAddOneEventInTwoWeeks.md) | Simple Calendar | Passed | 4m 33s | 24 |
| [Simple Calendar Add One Event Relative Day](https://vostride.com/benchmark/android-world/SimpleCalendarAddOneEventRelativeDay.md) | Simple Calendar | Passed | 3m 48s | 20 |
| [Simple Calendar Add One Event Tomorrow](https://vostride.com/benchmark/android-world/SimpleCalendarAddOneEventTomorrow.md) | Simple Calendar | Passed | 5m 30s | 25 |
| [Simple Calendar Add Repeating Event](https://vostride.com/benchmark/android-world/SimpleCalendarAddRepeatingEvent.md) | Simple Calendar | Passed | 3m 9s | 17 |
| [Simple Calendar Any Events On Date](https://vostride.com/benchmark/android-world/SimpleCalendarAnyEventsOnDate.md) | Simple Calendar | Passed | 2m 24s | 13 |
| [Simple Calendar Delete Events](https://vostride.com/benchmark/android-world/SimpleCalendarDeleteEvents.md) | Simple Calendar | Passed | 2m 12s | 12 |
| [Simple Calendar Delete Events On Relative Day](https://vostride.com/benchmark/android-world/SimpleCalendarDeleteEventsOnRelativeDay.md) | Simple Calendar | Passed | 2m 6s | 11 |
| [Simple Calendar Delete One Event](https://vostride.com/benchmark/android-world/SimpleCalendarDeleteOneEvent.md) | Simple Calendar | Passed | 1m 4s | 6 |
| [Simple Calendar Event On Date At Time](https://vostride.com/benchmark/android-world/SimpleCalendarEventOnDateAtTime.md) | Simple Calendar | Passed | 1m 1s | 4 |
| [Simple Calendar Events In Next Week](https://vostride.com/benchmark/android-world/SimpleCalendarEventsInNextWeek.md) | Simple Calendar | Passed | 3m 37s | 13 |
| [Simple Calendar Events In Time Range](https://vostride.com/benchmark/android-world/SimpleCalendarEventsInTimeRange.md) | Simple Calendar | Passed | 1m 59s | 9 |
| [Simple Calendar Events On Date](https://vostride.com/benchmark/android-world/SimpleCalendarEventsOnDate.md) | Simple Calendar | Passed | 2m 42s | 14 |
| [Simple Calendar First Event After Start Time](https://vostride.com/benchmark/android-world/SimpleCalendarFirstEventAfterStartTime.md) | Simple Calendar | Passed | 1m 35s | 6 |
| [Simple Calendar Location Of Event](https://vostride.com/benchmark/android-world/SimpleCalendarLocationOfEvent.md) | Simple Calendar | Passed | 57s | 5 |
| [Simple Calendar Next Event](https://vostride.com/benchmark/android-world/SimpleCalendarNextEvent.md) | Simple Calendar | Passed | 53s | 5 |
| [Simple Calendar Next Meeting With Person](https://vostride.com/benchmark/android-world/SimpleCalendarNextMeetingWithPerson.md) | Simple Calendar | Passed | 45s | 4 |
| [Simple Draw Pro Create Drawing](https://vostride.com/benchmark/android-world/SimpleDrawProCreateDrawing.md) | Simple Draw | Passed | 2m 6s | 10 |
| [Save Copy Of Receipt Task Eval](https://vostride.com/benchmark/android-world/SaveCopyOfReceiptTaskEval.md) | Simple Gallery | Passed | 1m 54s | 11 |
| [Simple Sms Reply](https://vostride.com/benchmark/android-world/SimpleSmsReply.md) | Simple SMS | Passed | 1m 13s | 6 |
| [Simple Sms Reply Most Recent](https://vostride.com/benchmark/android-world/SimpleSmsReplyMostRecent.md) | Simple SMS | Passed | 1m 10s | 6 |
| [Simple Sms Resend](https://vostride.com/benchmark/android-world/SimpleSmsResend.md) | Simple SMS | Passed | 2m 40s | 12 |
| [Simple Sms Send](https://vostride.com/benchmark/android-world/SimpleSmsSend.md) | Simple SMS | Passed | 1m 32s | 8 |
| [Simple Sms Send Clipboard Content](https://vostride.com/benchmark/android-world/SimpleSmsSendClipboardContent.md) | Simple SMS | Passed | 1m 41s | 9 |
| [Simple Sms Send Received Address](https://vostride.com/benchmark/android-world/SimpleSmsSendReceivedAddress.md) | Simple SMS | Passed | 1m 23s | 7 |
| [Tasks Completed Tasks For Date](https://vostride.com/benchmark/android-world/TasksCompletedTasksForDate.md) | Tasks | Passed | 3m 2s | 11 |
| [Tasks Due Next Week](https://vostride.com/benchmark/android-world/TasksDueNextWeek.md) | Tasks | Passed | 6m 50s | 38 |
| [Tasks Due On Date](https://vostride.com/benchmark/android-world/TasksDueOnDate.md) | Tasks | Passed | 1m 40s | 8 |
| [Tasks High Priority Tasks](https://vostride.com/benchmark/android-world/TasksHighPriorityTasks.md) | Tasks | Passed | 59s | 6 |
| [Tasks High Priority Tasks Due On Date](https://vostride.com/benchmark/android-world/TasksHighPriorityTasksDueOnDate.md) | Tasks | Passed | 4m 22s | 21 |
| [Tasks Incomplete Tasks On Date](https://vostride.com/benchmark/android-world/TasksIncompleteTasksOnDate.md) | Tasks | Passed | 1m 0s | 6 |
| [Vlc Create Playlist](https://vostride.com/benchmark/android-world/VlcCreatePlaylist.md) | VLC | Passed | 4m 17s | 22 |
| [Vlc Create Two Playlists](https://vostride.com/benchmark/android-world/VlcCreateTwoPlaylists.md) | VLC | Passed | 7m 48s | 44 |