On September 16, 2026, Google released "Android Bench 2.0," an AI evaluation framework for Android app development. The first generation measured localized fixes with a median of 32 lines across 1–2 files. Version 2.0 expands its scope to long-horizon tasks that would take a human engineer days to weeks, testing changes of up to 8,200 lines across 294 files.
The top-ranked combination, GPT 6 Astra paired with Codex, achieved an average completion rate of 82.2%. Yet only 28.0% of runs achieved a full pass. The completion rate isn't a measure of how much work or code was produced—it's a weighted score across functionality, regression, requirements, and visual criteria. Even when major screens and features work, some runs stall on state retention after rotation or on dependency issues, failing to fully polish lifecycle handling or secondary screens. What Android Bench 2.0 has started measuring isn't whether AI can write a large volume of code, but whether it can carry a product all the way to completion.
Even with high completion rates, the wall to a full pass remains
Android Bench 2.0's leaderboard displays pass rate and completion rate as separate metrics. Pass rate is the percentage of runs that met all functional tests and visual requirements, had no constraint violations, and scored a perfect 1.0. Completion rate, by contrast, is a continuous metric that sums the functionality, regression, requirements, and visual scores weighted per task, then applies a penalty multiplier for constraint violations.
Lining up the top four systems reveals that these two numbers don't move in the same direction.
| Model + Agent | Pass Rate | Avg. Completion Rate | Avg. Time | Avg. Cost (all 30 tasks) |
|---|---|---|---|---|
| GPT 6 Astra + Codex | 28.0% | 82.2% | 7.9 hours | $375.7 |
| Claude Fable 5.1 + Claude Code | 22.7% | 82.4% | 22.2 hours | $492.6 |
| GPT 5.6 Sol + Codex | 19.3% | 74.3% | 8.6 hours | $235.8 |
| Gemini 3.8 Flash + Antigravity SDK | 8.0% | 47.4% | 12.1 hours | $34.5 |
Fable 5.1 slightly edges out Astra on completion rate, but falls behind on pass rate. Astra's pass rate also carries a wide confidence interval of 13.3–42.0%. With only 30 tasks in the set, it's still too early to read small ranking differences as definitive gaps in ability.
Task-level results reveal what kind of "near misses" the completion rate captures. In Bitwarden's Navigation 3 migration, one run modified numerous files and passed the tests, yet ended with a visual snag—a missing screen transition at the end. In a news app's video playback feature, a player that scrolled off-screen failed to release properly, continuing to play in the background. Even when things look finished on the surface, defects that matter during actual operation can remain.
In other words, 82.2% doesn't mean "you can hand off this work with an 82.2% success rate." It's a number that compresses both the system's ability to advance major functionality and the residual risk that requires final human review into a single average.
Workload expanded from 32 lines to up to 8,200 lines
The original Android Bench was designed primarily around small-scale fixes to existing GitHub repositories. The median change size was 32 lines across 1–2 files, and leading models achieved pass rates around 91%. Google judged this state to represent a saturated benchmark.
Version 2.0 introduces 30 long-horizon tasks, requiring each system to perform 5 independent runs per task. New app creation involves 1,200–5,500 lines across 20–70 files. Library and architecture migrations span 200–8,200 lines across 5–294 files. There are also new feature additions and conversions from Flutter or React Native to native Android.
Directly linking the old version's roughly 91% to 2.0's 28.0% and concluding that AI capability has declined would be a mistake. The workload being measured, the scoring method, and the execution infrastructure all changed simultaneously. If anything, 28.0% is the result of making it possible to measure differences again that had been obscured by short fixes.
The strengths and weaknesses Google highlights can't be explained by code volume alone. Tasks with stable conversion rules—like Java to Kotlin, Retrofit to Ktor, or introducing a ViewModel layer—tend to scale well even across many files. On the other hand, breaking API changes, dependency injection gaps only discoverable at runtime, and unfamiliar new libraries with few training examples tend to cause systems to stall. The real difficulty lies not in length, but in the implicit connections within existing architecture.
Running isn't passing: multifaceted verification on an AVD
Each of the 30 tasks runs in a fresh Docker container every time. It uses a KVM-enabled Android Virtual Device (AVD), with environments allocated at least 16 CPUs, 72GB RAM, and 500GB of storage. Models don't just write commands in Markdown—they operate the shell through a structured API.
Verification happens in two layers. On the deterministic side, Instrumentation tests run, directly inspecting the contents of SQLite and Room. Intents, network calls, and Wear OS synchronization are also monitored, with regression tests confirming that existing functionality hasn't broken. This design prioritizes user-facing state changes over unit tests that merely fix the shape of internal classes.
Screens aren't judged by exact pixel matching. Differences in time display, battery icon, or font rendering are captured as noise, which could otherwise cause a semantically correct screen to fail. Instead, automated operation captures the screen and accessibility tree, and Gemini 3.5 Flash scores it from 0 to 1 against a reference image. Tricks like pasting a static image onto the screen to game the test are blocked by accessibility inspection, which reads touch targets and native elements.
Google states that across 360 calibration runs, this visual judgment matched 100% across repetitions. However, what matched was the output of the same judge model. This number doesn't reveal how well it aligns with human Android engineers, or whether it carries biases favoring specific model families.
Scoring isn't lenient partial credit, either. If a build fails, if a test is bypassed improperly, or if Flutter, Dart, or JavaScript files are reused in a nativization task, the multiplier applied to the completion rate becomes 0.0. Using legacy APIs in a Jetpack Compose task results in a 0.5x multiplier. Creating a large number of screens doesn't help if foundational constraints are violated—points are lost regardless.
The leaderboard compares systems, not models
Each row on the leaderboard isn't a standalone model. Google ran GPT 5.6 Sol with Codex, and Gemini 3.8 Flash with the Antigravity SDK. The Claude family is paired with Claude Code, and Qwen with Qwen Coder. Changing the agent harness changes the available tools, context compression, caching, and how failures are recovered from.
Google itself explains that prompt caching and short tool-specific context reduce token usage, and that agent harness design has a positive influence on results. This is close to real-world practice, since what developers buy isn't just model weights but a product that includes the execution infrastructure. But if you want to know "which model is smartest," this adds an extra variable.
Cost and time also can't be directly translated into an efficiency ranking. The official methodology acknowledges a bias where systems that fail early appear to have lower total execution time and cost. Network latency to the API gets mixed into wait times, and price changes disrupt cost comparisons made across different time periods. Provider-side prompt caching may also not be fully reflected in usage figures.
Therefore, Gemini 3.8 Flash at $34.5 can't be called a configuration that accomplishes the same work more cheaply than Astra at $375.7. Their completion rates differ significantly—47.4% versus 82.2%. To compare costs meaningfully, you'd first need to align systems with comparable pass rates and completion rates.
A mismatch in task classification between the methodology and the leaderboard
Android Bench 2.0 puts four measures in place against the problem of benchmark answers leaking into training data. New-creation tasks use internal apps with no public track record. Migration tasks use combinations that don't exist in upstream repositories, and conversion tasks use apps with no existing native Android version. On top of this, agent execution trajectories are audited to detect external code retrieval or score-gaming.
There's a tradeoff. While the framework is published on GitHub, the complete dataset of 30 tasks is not public, and Google says it's considering a publication method that avoids contamination. Third parties can inspect the skeleton of the environment but can't fully reproduce the ranking with the same tasks and weights. The stronger the contamination resistance, the harder external auditing becomes.
There's also a small but non-negligible discrepancy between official pages. The official methodology states the 30 tasks break down as 9 new-app-creation, 13 migration, 6 new-feature, and 2 conversion tasks, while the leaderboard's filter displays 10 new-creation, 13 migration, 5 new-feature, and 2 conversion—a one-task mismatch in both the new-creation and new-feature categories. As of the September 18 check, it's unclear whether this is a lag in page updates or a reclassification, so the definitions need to be aligned before reading category-specific results.
Other areas remain unmeasured as well. Bluetooth devices, physical cameras, and GPU shaders are simulated in software, and backends use local mocks. Foldable devices, large-screen devices, and Android Auto are targets for the future. This isn't a ranking that represents all of Android development.
Android Bench 2.0 is better read as material for designing your own review process than as a single table for picking one overall winner. Organizations centered on new development should look at creation tasks; those managing existing apps should look at the reasons behind migration and regression failures. Compare configurations with similar pass rates first, then layer in confidence intervals, average completion rate, time, and cost.
The human role changes too. If average completion rate exceeds 80% but full passes don't reach 30%, review efforts should shift focus away from stylistic preferences and toward state persistence and lifecycle handling. Dependencies and accessibility should also become priorities. A high completion rate isn't a signal that human involvement is unnecessary—it's a signal that tests and design reviews capable of catching the remaining defects are needed.
When the same model can be run across multiple execution harnesses, when the non-public tasks can be audited by third parties without contamination, and when the category definitions across official pages are consistent—once those conditions are met, Android Bench 2.0 will move closer to being a measurement tool for integrating AI into development workflows, rather than a promotional ranking of models.