Stop Writing Phone Automation Scripts! Use mobile-use Instead

Stop Writing Phone Automation Scripts! Use mobile-use Instead

What if you never had to write another brittle XPath selector or coordinate-based tap script again? What if your phone could simply understand what you want—and do it?

Every mobile developer and QA engineer knows the nightmare. You spend hours crafting automation scripts with Appium, only to watch them shatter when a button moves three pixels left. You maintain fragile UIAutomator hierarchies that break with every app update. You write hundreds of lines of code just to extract a simple list of emails. The mobile automation landscape is a graveyard of abandoned test suites and half-working prototypes.

But something just shifted. AI agents can now use real Android and iOS apps, just like a human.

Enter mobile-use—the open-source framework that's making traditional mobile automation obsolete. This isn't another wrapper around existing tools. This is a fundamentally different approach: an AI agent that sees your screen, understands your intent in plain English, and manipulates your device through the actual UI. No root access. No injected code. No coordinate hacking.

And here's the kicker: it's already achieved 100% completion on the AndroidWorld benchmark—the first framework ever to do so. While others brag about 70% or 80% success rates, mobile-use cracked the code on perfect mobile task automation.

Ready to see how? Let's dive into the future of phone automation.

What is mobile-use?

mobile-use is an open-source AI agent framework developed by Minitap, Inc. that enables natural language control of Android and iOS devices. Unlike traditional automation tools that rely on predefined selectors and scripted interactions, mobile-use employs a multi-agent architecture powered by large language models (LLMs) to perceive, reason about, and interact with mobile UIs dynamically.

The project emerged from cutting-edge research in multi-agent systems for mobile automation. Its creators—Pierre-Louis Favreau, Jean-Pierre Lo, and their team—published their findings in the paper "Do Multi-Agents Dream of Electric Screens? Achieving Perfect Accuracy on AndroidWorld Through Task Decomposition" (arXiv:2602.07787, 2026). This isn't hobbyist code; it's peer-reviewed science turned into production-ready tooling.

What makes mobile-use genuinely disruptive is its UI-aware automation capability. Traditional tools like Appium or UIAutomator require explicit knowledge of element hierarchies—IDs, classes, XPaths. mobile-use instead leverages accessibility trees and visual understanding to navigate interfaces the way a human would: reading labels, identifying buttons by context, and adapting when layouts change.

The framework is built in Python^{↗ Bright Coding Blog} 3.12+ and designed for extensibility. You can swap in different LLM providers—OpenAI, Google Vertex AI, or any OpenAI-compatible API endpoint including local models. This flexibility matters: you're not locked into a single vendor's pricing or availability.

Community momentum is accelerating fast. With active Discord development, rapid GitHub star growth, and a cloud platform already live at platform.mobile-use.ai, mobile-use is positioning itself as the default choice for next-generation mobile automation.

Key Features That Change Everything

🗣️ Natural Language Control

The headline feature is exactly what it sounds like: you tell your phone what to do in plain English, and it executes. "Open Gmail, find all unread emails from my boss, and summarize them." No coding. No selectors. Just intent. The LLM parses your command, decomposes it into sub-tasks, and orchestrates the execution through the UI.

📱 UI-Aware Automation

This is where mobile-use diverges radically from legacy tools. It doesn't need you to identify elements beforehand. It reads accessibility trees, interprets visual layouts, and makes contextual decisions about where to tap, scroll, or type. The system adapts when apps update, when themes change, when dynamic content loads. One caveat: games remain challenging since they typically don't expose accessibility tree data—though this is an industry-wide limitation, not unique to mobile-use.

📊 Intelligent Data Scraping

Extract structured data from any app using natural language descriptions of your desired output format. Want a JSON list of restaurant names and ratings from a food delivery app? Describe it. The agent navigates, collects, and formats—handling pagination, loading states, and variable layouts automatically.

🔧 Extensible LLM Configuration

You're not trapped with one provider. Configure OpenAI, Google Vertex AI, local LLMs via Ollama, or any OpenAI-compatible endpoint. The llm-config.override.jsonc system lets you assign different models to different agent nodes—use a cheap fast model for simple decisions, an expensive capable model for complex reasoning.

🏆 Benchmark-Driven Excellence

The AndroidWorld 100% score isn't marketing fluff. AndroidWorld tests real-world task completion across diverse apps and scenarios. Achieving perfect accuracy required solving fundamental challenges in error recovery, context maintenance, and multi-step planning that plague other frameworks.

Real-World Use Cases Where mobile-use Dominates

1. Automated QA and Regression Testing

Traditional mobile test automation requires massive upfront investment in page objects, element maps, and maintenance overhead. With mobile-use, QA teams write test scenarios in English: "Purchase a $5 gift card, verify confirmation screen shows correct amount, check email receipt arrives within 60 seconds." The agent executes across devices, adapts to UI variations between OS versions, and generates structured pass/fail reports with extracted evidence.

2. Competitive Intelligence and Market Research

Need to monitor pricing across food delivery apps? Track feature rollouts in competitor apps? mobile-use agents can systematically navigate third-party applications, extract structured data, and compile reports—all without API access or reverse engineering. The --output-description parameter ensures you get exactly the data format you need for downstream analysis.

3. Accessibility Testing and Compliance

Verify that your app works with assistive technologies by having mobile-use execute real user journeys through screen reader paths. The framework's reliance on accessibility trees means it naturally validates that your UI elements are properly labeled and navigable.

4. Personal Productivity Automation

Power users can automate repetitive phone tasks without learning automation frameworks. "Every morning, check calendar, open weather app, send myself a text with today's meetings and forecast." The Docker^{↗ Bright Coding Blog} quickstart makes this accessible even to non-developers.

5. Customer Support Simulation

Train support staff and test help documentation by having mobile-use execute exact customer-reported reproduction steps. Documented workflows become executable, verifiable, and automatically updated when apps change.

Step-by-Step Installation & Setup Guide

Quick Start: Docker (Recommended for Android)

The fastest path to automation, requiring only Docker and an Android device or emulator:

Prerequisites:

• Docker installed and running
• Android device with USB debugging enabled, OR Android Studio emulator running
• Device and computer on same Wi-Fi network

Linux/macOS:

# Make script executable
chmod +x mobile-use.sh

# Run your first automation
bash ./mobile-use.sh \
  "Open Gmail, find first 3 unread emails, and list their sender and subject line" \
  --output-description "A JSON list of objects, each with 'sender' and 'subject' keys"

Windows (PowerShell):

powershell.exe -ExecutionPolicy Bypass -File mobile-use.ps1 `
  "Open Gmail, find first 3 unread emails, and list their sender and subject line" `
  --output-description "A JSON list of objects, each with 'sender' and 'subject' keys"

Critical: Accept any ADB connection prompts on your device. The script connects via IP, so both devices must share a network.

Manual Development Setup

For customization, debugging, or iOS simulator support:

1. Clone and enter repository:

git clone https://github.com/minitap-ai/mobile-use.git && cd mobile-use

2. Configure environment:

# Copy template and add your API keys
cp .env.example .env
# Edit .env with your OPENAI_API_KEY or other provider credentials

3. Configure LLM (optional but recommended):

cp llm-config.override.template.jsonc llm-config.override.jsonc
# Edit to specify models, providers, and routing between agent nodes

For Google Vertex AI specifically:

• Ensure gcloud credentials are configured, OR
• Set GOOGLE_APPLICATION_CREDENTIALS to your service account JSON path

4. Create Python environment:

# Requires uv: https://github.com/astral-sh/uv
uv venv                    # Creates .venv with Python 3.12+
source .venv/bin/activate  # macOS/Linux
# OR: .venv\Scripts\activate  # Windows

5. Install dependencies:

uv sync  # Reproducible install from locked requirements

Platform-Specific Requirements:

Troubleshooting Common Issues

REAL Code Examples from mobile-use

Let's examine actual usage patterns from the repository, with detailed explanations of what happens under the hood.

Example 1: Basic Natural Language Command

python ./src/mobile_use/main.py "Go to settings and tell me my current battery level"

What's happening here: This is the simplest invocation pattern. The main.py entry point receives a natural language string, which gets processed through the multi-agent pipeline shown in the architecture diagram. The Planner Agent decomposes this into sub-tasks: (1) open Settings app, (2) navigate to Battery section, (3) extract percentage value, (4) formulate natural language response. The Executor Agent then drives the actual UI interactions—tapping icons, scrolling if needed, reading text values from the accessibility tree. Finally, the Reporter Agent compiles the extracted information into a human-readable answer.

Notice there's no device specification in this command—mobile-use auto-detects available devices via ADB or idb, preferring the first connected Android device or active iOS simulator.

Example 2: Structured Data Extraction

python ./src/mobile_use/main.py \
  "Open Gmail, find all unread emails, and list their sender and subject line" \
  --output-description "A JSON list of objects, each with 'sender' and 'subject' keys"

This is where mobile-use gets powerful. The --output-description parameter triggers the Structured Output Agent, which modifies the LLM's behavior to enforce valid JSON schema compliance. Here's the execution flow:

1. Navigation Phase: The agent opens Gmail, handles any onboarding screens or permissions, and locates the inbox
2. Detection Phase: It identifies unread email indicators (bold text, specific icons, "unread" labels) using the accessibility tree
3. Extraction Phase: For each unread email, it reads sender name and subject line text
4. Transformation Phase: The LLM formats extracted data into the requested JSON structure, with validation against the description
5. Pagination Handling: If emails span multiple screens, the agent scrolls and repeats extraction until satisfied all items are captured

The JSON description acts as a soft schema—the LLM interprets your intent and generates appropriately structured output, without rigid predefined fields.

Example 3: LLM Configuration Override

// llm-config.override.jsonc - customize which models power each agent
{
  "planner": {
    "provider": "openai",
    "model": "gpt-4o",
    "temperature": 0.2  // Low temperature for consistent, deterministic planning
  },
  "executor": {
    "provider": "openai", 
    "model": "gpt-4o-mini",
    "temperature": 0.1   // Even lower for reliable UI element selection
  },
  "reporter": {
    "provider": "vertex_ai",
    "model": "gemini-1.5-pro",
    "temperature": 0.7   // Higher for natural, varied response language
  }
}

Advanced configuration explained: This demonstrates the multi-model routing capability. Different agent nodes have different requirements:

• Planner needs strong reasoning and context window—worth the GPT-4o cost
• Executor makes rapid, repetitive decisions—GPT-4o-mini is faster and cheaper
• Reporter benefits from Gemini's long-context capabilities for summarization

The .jsonc format allows comments for documentation. Environment variables like OPENAI_BASE_URL enable local LLM deployment through tools like Ollama or vLLM—critical for air-gapped environments or cost optimization at scale.

Example 4: Docker Script Deep-Dive

bash ./mobile-use.sh \
  "Open Gmail, find first 3 unread emails, and list their sender and subject line" \
  --output-description "A JSON list of objects, each with 'sender' and 'subject' keys"

Behind the scenes: This shell script orchestrates several complex operations:

1. Device Discovery: Runs adb devices to find USB-connected devices, then adb shell ip addr to determine Wi-Fi IP
2. Network Bridge: Configures ADB TCP/IP mode (adb tcpip 5555) so the Docker container can connect wirelessly
3. Container Launch: Pulls the pre-built image from GHCR with all dependencies pre-installed
4. Volume Mounting: Injects your .env and llm-config.override.jsonc into the container
5. Execution: Runs the Python main module with your arguments inside the isolated environment
6. Cleanup: Restores USB mode (adb usb) on script exit

The --interface fallback exists because Android devices use varying WLAN interface names (wlan0, eth0, ap0) depending on manufacturer and Android version.

Advanced Usage & Best Practices

Optimize Costs with Model Tiers: Not every agent needs frontier models. Configure GPT-4o-mini or local Llama 3.1 for executor tasks, reserving expensive models for planning and error recovery. Monitor token usage through your provider dashboards—mobile-use's modular architecture makes A/B testing trivial.

Handle Flaky Networks Gracefully: The Docker approach's Wi-Fi dependency introduces fragility. For CI/CD pipelines, prefer Android emulators with host network visibility, or implement retry logic around the shell script with exponential backoff.

Secure Your Credentials: Never commit .env files. The repository includes .env.example specifically to prevent accidental key exposure. For team deployments, consider secret management integration—HashiCorp Vault, AWS^{↗ Bright Coding Blog} Secrets Manager, or GitHub Actions encrypted variables.

Extend with Custom Agents: The architecture diagram in doc/graph.png reveals the system's composability. Advanced users can inject custom nodes into the agent graph—perhaps a Screenshot Verification Agent that uses CLIP embeddings to validate visual states, or a Performance Monitoring Agent that tracks execution latency per step.

Benchmark Your Use Case: AndroidWorld scores don't guarantee your specific app works perfectly. Start with a representative task suite, measure success rates, and iterate on prompts and model configurations. The --output-description parameter is particularly sensitive to phrasing precision.

Comparison with Alternatives

The Verdict: Choose mobile-use when you need rapid deployment without selector maintenance, when tasks require reasoning and adaptation, or when you want to leverage LLM capabilities natively. Stick with Appium or Maestro for simple, stable apps where speed and determinism outweigh flexibility, or where team expertise already exists.

FAQ

Q: Does mobile-use require rooting my Android phone or jailbreaking iOS?

A: Absolutely not. It operates through standard accessibility services and ADB (Android) or idb (iOS simulators)—the same mechanisms used by legitimate automation and testing tools.

Q: Can I use mobile-use with physical iOS devices?

A: Not yet. Physical iOS support is on the roadmap. Currently, iOS simulators on macOS are fully supported through Xcode and Facebook's idb companion.

Q: What LLM providers work out of the box?

A: OpenAI, Google Vertex AI, and any OpenAI API-compatible endpoint—including local deployments via Ollama, LM Studio, or vLLM. Configure via environment variables and llm-config.override.jsonc.

Q: How does mobile-use handle app updates that change UI layouts?

A: This is its core advantage. Because it reasons about UI semantically rather than relying on fixed selectors, it often adapts automatically to layout changes, theme updates, and reorganized menus.

Q: Is the cloud platform free to try?

A: Yes—visit platform.mobile-use.ai for managed execution without local setup. Pricing details are available on the platform.

Q: How do I cite mobile-use in academic work?

A: Use the provided BibTeX entry from the README, or the formatted citation: Favreau et al., "Do Multi-Agents Dream of Electric Screens?", arXiv:2602.07787 (2026).

Q: What's the performance overhead compared to native automation?

A: LLM inference adds latency—typically 1-5 seconds per decision depending on model and provider. For most automation scenarios, this is acceptable. For sub-second requirements, consider hybrid approaches with cached planning.

Conclusion

The era of writing and maintaining brittle mobile automation scripts is ending. mobile-use represents a paradigm shift—from explicit programming to intent-based delegation. Its 100% AndroidWorld score proves this isn't speculative; it's production-ready superiority.

For developers, QA engineers, researchers, and power users, the value proposition is stark: describe what you want, get working automation. No selectors. No maintenance nightmares. No vendor lock-in.

The framework is evolving rapidly. Join the Discord community, contribute on GitHub, or dive straight in with the Docker quickstart. The future of phone automation isn't writing more code—it's speaking your mind and letting AI handle the rest.

⭐ Star the repository, try the cloud platform, and stop writing phone automation scripts forever.