MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks

The Problem: Benchmarks Miss How Mobile Agents Actually Work

Existing benchmarks for mobile AI agents fall into two traps. GUI-focused ones test whether an agent can tap the right button on screen—superficial interactions that ignore the real work happening in the background, like accessing databases or managing long chains of decisions. Function-calling benchmarks, meanwhile, test agents in sterile offline environments where they match function names to descriptions, divorced from the messy reality of actual app ecosystems with live data, runtime errors, and permission constraints. Neither captures what a real mobile copilot needs to do.

MobilePA-Bench closes this gap with an interactive, stateful benchmark that runs agents against executable mobile sandboxes—not just screenshots or pretend APIs. The benchmark spans 13 functional domains (calendar, messaging, notes, etc.) and 212 realistic mobile tools, evaluating whether agents can actually accomplish complex, real-world tasks.

What Gets Tested: Three Advanced Capabilities

Beyond simple tool calls, the benchmark measures three capabilities that separate competent agents from unreliable ones.

Sub-agent Collaboration requires agents to decompose complex tasks and delegate work to specialized sub-agents. Rather than handling everything monolithically, an agent must recognize when to route a request to a calendar expert or messaging specialist.

Memory Usage tests whether agents can recall stored information—user preferences, past interactions, saved profiles—to interpret vague requests. "Book my usual coffee slot" demands retrieving what "usual" means from persistent memory.

Skill Usage evaluates whether agents invoke pre-built composite skills (e.g., "send meeting reminder") instead of re-planning every atomic step, reducing errors and latency.

Why This Matters: Current LLMs Fail in Real Conditions

Experiments reveal a sobering reality: frontier LLMs—the most advanced models available—perform reliably in lab conditions but crumble under real mobile constraints. Performance drops sharply when:

• Tool ordering is strict (steps must execute in a specific sequence)
• Permissions limit what tools can access
• Runtime errors occur unpredictably

This gap between controlled benchmarks and actual deployment exposes why existing evaluations are misleading. An agent that scores well on offline function-calling might fail catastrophically when deployed on a real phone where a database query times out or a permission is denied mid-task.

The Technical Innovation

MobilePA-Bench pairs an interactive function-calling sandbox—maintaining live application databases and returning structured feedback—with evidence-based verification. This dual approach creates both a diagnostic tool (identifying where and why agents fail) and an interactive foundation for reinforcement learning (using failures as signals to improve agent policies).

The benchmark doesn't rely on static, offline matching. It executes tool calls against actual backend simulators, catching errors and constraint violations that no paper-and-pencil test would reveal.

Bottom Line

MobilePA-Bench exposes that we've been measuring the wrong things. By enforcing real-world conditions—stateful execution, permission limits, runtime failures—it reveals how fragile current mobile agents actually are and provides a rigorous foundation for building dependable ones. For anyone shipping agent-based mobile apps, this benchmark is essential diagnostics.