Google's ARTEMIS: Natural-Language Automation for Real Android Phones

Phones have been the odd one out in the rise of AI agents. On a desktop or in a browser, agents increasingly click through interfaces, drive applications, write code, read files and follow plain-language instructions with little supervision. On a handset the story has been far less impressive. Conventional test frameworks tend to be brittle, lean hard on fixed element identifiers, and snap the moment a layout changes. Worse, none of them were designed for an agent that can simply be told a goal and work out the steps itself.

The gap stands out because the smartphone is where most people spend most of their digital time. Software that handles computers and browsers well still struggles to operate a real phone.

ARTEMIS, a project Google has just open-sourced, attacks the problem differently. Developers don't have to describe every screen, widget and interaction in advance. An AI agent receives an instruction in natural language and carries it out on a real Android device or an emulator. The project reports completing more than 99 percent of the tasks on a demanding multi-step benchmark, and it plugs into coding agents developers already run, among them Codex, Claude Code and Antigravity.

Even more intriguing is the direction on the roadmap: fully local execution on the device, driven by lightweight vision models, so that not every interaction has to travel to a cloud service. If that lands, mobile automation could shift from a fragile pile of scripts toward the agent-style interfaces already familiar from desktops and browsers, something that grasps what needs doing and operates the device to get it done.

Scripts versus an agent that looks

Classic mobile automation asks the test author to name each element a script touches, usually through an identifier compiled into the app, and to spell out a rigid sequence of actions ahead of time. That works acceptably on a stable screen inside an app whose internals the author can see and control. It collapses when anything moves. A redesigned button, a renamed identifier, or an unfamiliar custom-drawn interface from a modern UI toolkit can quietly break a script that ran reliably for months, and working out why generally sends a person back through the failure by hand.

An AI-driven design turns that model around. Rather than replaying hard-coded steps tied to identifiers, the agent looks at what the screen currently shows, interprets what the instruction means, and picks the next action on the spot, much like someone opening an unfamiliar app for the first time. When a button has moved, changed its label or been redesigned altogether, the agent adjusts to what's in front of it instead of failing on a reference to something that no longer exists.

One sentence in place of a script

The payoff is easiest to see when a single instruction replaces what would otherwise be a long, hand-written test. Instead of enumerating taps, swipes and element lookups, a task can be phrased the way it would be explained to a colleague:

Open Settings, find Battery and tell me current level

Given that request, the agent moves through the real phone interface. It finds the right screen by combining knowledge of how the app is organized with a visual reading of what's displayed, pulls out the requested value and reports it back. Nobody wrote a line of traditional automation code aimed at particular screen elements.

The same approach scales to more involved workflows that cross several apps. One instruction can describe a sequence hopping between unrelated applications, which a conventional script would usually need as several separate, carefully coordinated pieces. Reasoning about intent, not executing a fixed script, is what makes that cross-app flexibility realistic without enormous up-front engineering for each workflow.

The diagram above, taken from the project's repository, shows an autonomous testing workflow in which Antigravity works together with ARTEMIS.

A benchmark result that can be checked

Bold automation claims are everywhere, and claims measured against a recognized independent yardstick are much rarer. ARTEMIS reports completing more than 99 percent of the tasks on a benchmark built specifically to evaluate mobile agents across many real applications and multi-step scenarios, not a small hand-picked set of easy examples.

The chart comes from the project's page and shows the 99-plus percent completion rate on AndroidWorld, a Google Research benchmark covering more than 20 apps and over 100 multi-step tasks.

A benchmark spanning that many applications and tasks is designed to stress the very things hard-coded automation struggles with: unfamiliar layouts, tasks that need several actions chained correctly, and interfaces nobody built with automation in mind. Clearing it at better than 99 percent represents a different tier of reliability from earlier AI-driven tools, which historically stumbled on open-ended, multi-step work in real, unmodified apps.

Connecting through MCP

ARTEMIS isn't a standalone tool with its own isolated workflow. It hooks into the coding agents already used every day, including Codex, Claude Code and Antigravity, through the Model Context Protocol, or MCP. That's an open standard that lets an AI agent discover and call outside tools consistently, whichever tool or model happens to be driving.

Since the link relies on an open protocol rather than a one-off connection to a single coding product, any coding agent that speaks MCP can control a real phone as one more tool in its kit, next to its existing abilities to read code, run commands or browse the web. Setup generally means registering the automation server in the coding agent's own configuration, in a shape much like adding any other external tool:

{
  "mcpServers": {
    "mobile-automation": {
      "command": "/path/to/tool/.venv/bin/python",
      "args": ["-m", "mcp_server"],
      "cwd": "/path/to/tool"
    }
  }
}

After that, a developer can describe a mobile testing job in plain language inside the coding agent's chat, the same way a code change would be described, and the agent drives the attached device directly. No switch to a separate testing application is needed.

Whole development loops on a device

The benefit reaches well past quick one-off checks such as reading a setting. A coding agent wired to a real device can run much bigger development flows by itself:

• build the newest version of an application
• install that fresh build on a connected physical device
• step through one particular user journey, say signing in using a test account
• look for unexpected behavior, for instance a popup that shouldn't appear after login
• return screenshots as visual evidence of what happened at each stage

The screenshot above shows a live demo from the project: planning driving routes and totaling their durations in Google Maps, then opening YouTube.

That kind of end-to-end flow, covering building, installing, navigating and verifying, mirrors what a human QA engineer does when checking a change before release, with an AI agent following natural-language instructions in place of a person tapping a physical device. Teams that already lean on agents across much of their development process can now push that agent-driven approach onto real hardware, which removes one of the most stubbornly manual steps left in an otherwise increasingly automated pipeline.

Knowing why a run failed

Driving a device is only half of a useful testing tool. When something breaks, understanding why matters as much as triggering the interaction. Alongside the on-screen actions, ARTEMIS captures diagnostics from the device while a workflow runs, including system logs and screenshots of the screen at meaningful moments.

Those captures matter for real debugging, not just happy-path automation. A test that fails halfway through a long sequence is far more useful when it returns a screenshot of the screen at the moment of failure, plus system log output from that same instant, than as a bare pass or fail verdict. A failed run becomes a practical starting point for investigation, not a dead end that forces the whole scenario to be reproduced manually just to see what happened.

The roadmap: private and local

The current capability is already a big step beyond classic mobile automation, yet the published roadmap points to something arguably bigger over time: execution on the device itself, powered by lightweight vision-language models running locally, instead of a constant connection to a cloud-hosted model for every action.

Two practical gains come with that shift. The first is latency. When reasoning and execution happen entirely on the phone, with no network round trip to a remote model per step, responses can arrive much faster, which matters for automation that has to keep up with an interactive, real-time interface instead of pausing before every action.

The second, arguably weightier gain is privacy. Automation that never ships a screenshot, or any other record of what's on screen, off the device is far better suited to sensitive material, whether personal data, confidential business information or anything else a person or company would rather keep on their own hardware.

Seen against the trend already well underway on desktops and in browsers, often called computer use, the roadmap carries that idea into the mobile world, with a deliberate emphasis on making it private and local rather than permanently dependent on the cloud. Desktop equivalents still generally route screen understanding through a hosted model, so a plan built around lightweight on-device vision models suggests a more self-contained path for phones.

What this changes for mobile teams

Taken together, natural-language control of a phone, a verifiable high-reliability benchmark result, integration with popular coding agents, built-in diagnostics and a roadmap aimed at local execution amount to something new for anyone who builds or tests mobile apps.

• Exploratory testing, which used to need a person poking around an app for surprises, can be phrased as one instruction and run automatically.
• Regression checks, which demanded carefully maintained brittle scripts tied to element identifiers, can be written in plain language while the agent adapts as the interface evolves.
• Development workflows that once ended at a code change, leaving on-device verification to a human, can run all the way through an automated build, install and verify cycle handled by the same agent that already assists the rest of the work.

None of this removes the value of a skilled tester's judgment, especially for subtle, subjective quality questions beyond whether an action technically succeeded. What changes is where human attention goes: far less on maintaining brittle hard-coded scripts against an interface that keeps shifting, and far more on deciding what matters to test, then letting an agent that reasons about intent carry it out.

Where it leaves mobile automation

Mobile automation has trailed behind what agents already manage on desktops and in browsers for years. ARTEMIS narrows the gap by letting AI agents control a real device through natural-language instructions. Its reported completion rate above 99 percent on a demanding independent benchmark is notable, and so is its direct integration with the coding agents developers already use daily.

The more interesting story may lie ahead. The roadmap aims at fully local, on-device execution powered by lightweight vision models. If that arrives, mobile agents would be more than capable. They'd be private and self-contained, able to work without sending any interaction to a remote service.

The wider significance reaches beyond automating taps, swipes or single test cases. It suggests a future where a real smartphone is one more environment an agent understands and operates, next to the terminal, the filesystem, the browser and the codebase. For people who build, test and maintain mobile apps, that could substantially change what an AI coding agent is able to do. The code lives in Google's public ARTEMIS repository on GitHub.