Google’s Android Bench 2.0 now tests AI coding agents on Android work that can take a human engineer several days or a week, rather than limiting the benchmark to small bug fixes and local code changes. The first results are less flattering: the best long-horizon pass rate is about 28%, compared with roughly 91% on the original task set.
Google published Android Bench 2.0 on 16 September, adding 30 long-horizon tasks across app creation, migrations, new features and app conversions. The tasks include building a multi-screen food-delivery app from design mocks, moving libraries and architecture, adding features such as widgets and Picture-in-Picture, and converting Flutter or React Native apps to native Android.
Android Bench 2.0 measures partial progress, not just failure
The benchmark now reports both pass rate and completion rate. A task that compiles, passes most functional checks and gets much of the UI right can still fail the full test, but its partial work is recorded instead of being flattened into a zero.
Google combines functionality, regression checks, requirements and visual fidelity in the completion score. The methodology also includes runtime checks, database inspection, accessibility-tree analysis and a multimodal visual judge. That is a better match for mobile software than checking whether a patch merely builds.
Where AI coding agents still stall
Google says models handle new code better than changes to complex existing systems. Repetitive migrations can scale across more than 125 files and 8,000 lines of code, while runtime validation, breaking framework changes and unreleased libraries cause more trouble. Cross-platform conversions remain particularly difficult, with frontier models reaching at most an 80% completion rate and no model achieving a perfect pass rate.
The leaderboard also pairs models with agents from their providers. Google tested GPT 5.6 Sol with Codex and Gemini 3.8 Flash with Antigravity, arguing that the harness around a model affects token use and developer outcomes. OpenAI’s GPT-6 Astra leads the new long-horizon table with a 28% pass rate, while Gemini 3.8 Flash records 8%.
Developers choosing an agent still have to account for the unfinished work. A model can produce a convincing first 80% of an Android app and leave the architectural cleanup, edge cases and visual polish to a human.
The benchmark is available through Google’s Android Bench leaderboard, while the methodology explains how its continuous completion rate is calculated. For related coverage, see how OpenAI put the Codex harness behind its developer API and why one new AI model wants to replace your senior developer.
What is Android Bench 2.0?
Android Bench 2.0 is Google’s benchmark for testing AI models and coding agents on complex Android engineering tasks, including app creation, migrations, new features and cross-platform conversions.
What is the highest Android Bench 2.0 pass rate?
OpenAI’s GPT-6 Astra has the highest reported long-horizon pass rate at 28%, according to Google’s Android Bench 2.0 announcement.
Why did Google add a completion rate?
The completion rate records partial progress on tasks that are not fully solved, so an agent that completes most of a large migration is not scored the same as one that never produces a working build.
