OpenAdapt: Automate Any GUI With Vision AI (No Code)

OpenAdapt: Automate Any GUI With Vision AI (No Code)

What if you could train an AI to use your computer just by showing it what to do? No brittle selectors. No XPath nightmares. No spending three days debugging why a button click broke after a UI update.

Here's the painful truth every developer and automation engineer knows: traditional RPA tools are dying. They're expensive, fragile, and require you to treat every interface like a rigid puzzle where one moved pixel means catastrophic failure. Meanwhile, Large Multimodal Models (LMMs) can see your screen, understand what they're looking at, and decide what to do next. The gap between "AI can do this" and "I can actually deploy this" has been massive—until now.

Enter OpenAdapt, the open-source bridge between cutting-edge vision-language models and the messy reality of desktop and web GUIs. This isn't another wrapper around Selenium or Playwright. It's a fundamentally different approach: demonstrate once, automate forever. Record your screen performing any task, feed that demonstration to a multimodal model like Claude, GPT-4o, or Qwen3-VL, and watch the AI execute with human-like visual understanding.

The results are already staggering. On controlled benchmarks, OpenAdapt's demo-conditioned prompting achieved 100% first-action accuracy—up from 46.7% zero-shot baseline. That's not a marginal improvement. That's the difference between "barely works" and "production-ready."

Ready to stop writing automation scripts that break every Tuesday? Let's dive into how OpenAdapt is rewriting the rules of process automation.

What Is OpenAdapt?

OpenAdapt is open-source Generative Process Automation—what the creators call "Generative RPA." It's the software adapter that connects Large Multimodal Models (LMMs), Large Action Models (LAMs), and Visual Language Models (VLMs) to traditional desktop and web graphical interfaces.

Born from the recognition that zero-shot vision models fail on GUI tasks due to ambiguity in UI affordances, OpenAdapt solves this through trajectory-conditioned disambiguation. In plain terms: instead of telling the AI "click the settings button," you show it exactly how you navigate there. The model conditions its decisions on your actual demonstration, resolving ambiguity that would otherwise cause failures.

The project follows a modular meta-package architecture in v1.0+, splitting functionality across focused PyPI packages. The core openadapt package provides a unified CLI, while specialized sub-packages handle capture, ML training, evaluation, privacy scrubbing, UI grounding, and more. This modularity means you install only what you need—no bloat, no unnecessary dependencies.

OpenAdapt is trending now because it arrives at a critical inflection point. Commercial tools like Adept, Rabbit, and various "AI agents" promise similar capabilities but lock you into proprietary ecosystems. OpenAdapt gives you the same underlying technology—LMM-powered GUI automation—with full source code, MIT licensing, and the ability to self-host. When OpenCUA (a NeurIPS 2025 Spotlight project from XLANG Lab) needed macOS accessibility capture code, they reused OpenAdapt's implementation. That kind of research validation signals where the field is heading.

The project's three-phase pipeline—Demonstrate → Learn → Execute—mirrors how humans actually teach each other. You show, the learner observes patterns, then applies that knowledge to new situations. It's intuitive, powerful, and fundamentally more robust than rule-based automation.

Key Features That Separate OpenAdapt From the Pack

Multimodal Demonstration Recording OpenAdapt captures not just screenshots but the full observation-action trajectory: mouse movements, clicks, keyboard input, scroll events, and the accessibility tree structure of applications. The openadapt-capture package handles this with minimal overhead, storing everything in a searchable demonstration library.

Privacy-First by Design Before any demonstration enters your training pipeline, openadapt-privacy scrubs PII and PHI. This isn't an afterthought—it's a core architectural concern. For healthcare, finance, or any regulated industry, this built-in anonymization is non-negotiable.

Demo-Conditioned Inference Here's OpenAdapt's secret weapon. While competitors use demonstrations only for training, OpenAdapt conditions runtime inference on retrieved demonstrations. The model sees your current screen plus relevant past demonstrations, dramatically improving accuracy on unfamiliar interfaces. The validated result speaks volumes: 100% first-action accuracy versus 46.7% zero-shot.

Policy/Grounding Separation The architecture cleanly separates what to do (Policy, decided by VLMs) from where to do it (Grounding, handled by openadapt-grounding). This modularity means you can swap in different models for each component—Claude for high-level reasoning, specialized grounding models for pixel-precise localization.

Safety Gates and Human-in-the-Loop Production automation without safety rails is reckless. OpenAdapt includes runtime validation layers, confirm modes for high-risk actions, and integration with openadapt-crier for Telegram-based approval workflows. The AI suggests; you approve—until you're confident enough to let it run autonomously.

Evaluation-Driven Improvement Success isn't assumed; it's measured. openadapt-evals provides standardized benchmarks, and successful execution traces automatically feed back as new training data. Your automation system literally gets smarter with every task it completes.

Modular, Extensible Ecosystem With 15+ focused packages—from openadapt-wright for developer automation to openadapt-consilium for multi-model consensus—you compose exactly the stack you need. No monolithic framework forcing its choices on you.

Real-World Use Cases Where OpenAdapt Dominates

Legacy System Automation That internal tool from 2008 with no API? The one that requires clicking through seventeen modal dialogs? OpenAdapt doesn't care. It sees the interface, references your demonstration, and executes. No API integration needed, no database access required. Pure visual automation for systems that will never get modernized.

Cross-Application Workflows Imagine onboarding a new employee: create Active Directory entry in one tool, provision email in another, set up project access in a third, send welcome message in Slack. Traditional RPA chokes on context switching. OpenAdapt treats the entire desktop as its environment, seamlessly moving between applications just like a human would.

Dynamic Web Application Testing Modern SPAs with shadow DOM, canvas-rendered charts, and unpredictable loading states break conventional testing tools. OpenAdapt's visual understanding doesn't depend on DOM structure. It recognizes "the blue export button" regardless of whether it's a <button>, a <div>, or a canvas-drawn element.

Regulated Industry Data Entry Healthcare claims processing, insurance underwriting, financial compliance reporting—all involve structured data entry across complex forms with strict accuracy requirements. Record a demonstrator performing the task correctly, scrub sensitive data with openadapt-privacy, then deploy agents that replicate that exact behavior with audit trails.

Accessibility Testing and Remediation By leveraging the accessibility tree capture alongside visual information, OpenAdapt can verify that applications are actually usable with assistive technologies, not just technically compliant. The openadapt-grounding package's UI element localization helps identify where focus management and screen reader announcements break down.

Step-by-Step Installation & Setup Guide

Getting started with OpenAdapt takes minutes, not hours. The modular installation lets you start minimal and expand as needed.

Prerequisites

You'll need Python^{↗ Bright Coding Blog} 3.10 or newer. Verify your version:

python --version
# Must show 3.10.x or higher

Platform-Specific Permissions

macOS users: Grant three critical permissions to your terminal application:

• Accessibility: Allows reading UI element structure
• Screen Recording: Enables screenshot capture for visual understanding
• Input Monitoring: Permits recording of keyboard and mouse actions

Navigate to System Preferences → Security & Privacy → Privacy → [Permission Type], then add your terminal emulator.

Windows users: Run your terminal as Administrator when capturing input events. This ensures reliable mouse and keyboard recording.

Installing OpenAdapt

Choose your installation path based on your needs:

# Minimal installation - CLI only, no capture or ML
pip install openadapt

# Include GUI capture and recording capabilities
pip install openadapt[capture]

# Add machine learning training and inference
pip install openadapt[ml]

# Include benchmark evaluation tools
pip install openadapt[evals]

# Add privacy scrubbing for PII/PHI
pip install openadapt[privacy]

# Install everything for full functionality
pip install openadapt[all]

For development work on sub-packages, clone and install in editable mode:

git clone https://github.com/OpenAdaptAI/openadapt-ml
cd openadapt-ml
pip install -e ".[dev]"

Verifying Installation

Run the built-in system check:

openadapt doctor

This validates your Python version, required permissions, and available sub-packages. Address any warnings before proceeding.

Check installed versions across the ecosystem:

openadapt version

REAL Code Examples: From Recording to Autonomous Execution

Let's walk through OpenAdapt's actual CLI usage, drawn directly from the repository documentation. These aren't toy examples—they're the commands you'll use in production.

Example 1: Recording Your First Demonstration

The foundation of everything in OpenAdapt is the capture. Here's how you record a task:

# Start recording with a descriptive name
openadapt capture start --name invoice-processing

# Perform your normal workflow in any GUI application:
# - Open your accounting software
# - Navigate to unpaid invoices
# - Select the first invoice
# - Click "Process Payment"
# - Enter payment details
# - Save and confirm

# Press Ctrl+C in the terminal to stop recording

What's happening under the hood: openadapt-capture subscribes to system-level input events and takes periodic screenshots. Each action is timestamped and correlated with the visual state. The accessibility tree (on macOS via atomacos, Windows via UI Automation) provides structured element information alongside raw pixels. This multimodal recording becomes your demonstration trajectory.

Example 2: Training a Vision-Language Model

Once you have demonstrations, train a model to replicate them:

# Train on your capture using Qwen3-VL 2B parameter model
openadapt train start --capture invoice-processing --model qwen3vl-2b

# Monitor training progress in another terminal
openadapt train status

# Stop training if needed (graceful checkpoint saving)
openadapt train stop

The technical depth: OpenAdapt loads your demonstration trajectories and fine-tunes the vision-language model using a custom loss function that emphasizes action prediction conditioned on visual state and demonstration context. The --model flag accepts various architectures—Qwen3-VL for efficiency, larger models for complex reasoning. Training outputs checkpoints to training_output/model.pt by default.

Example 3: Evaluating Trained Agents

Before deploying, validate your model's performance:

# Evaluate your trained checkpoint against a benchmark
openadapt eval run --checkpoint training_output/model.pt --benchmark waa

# Or evaluate an API-based agent without local training
openadapt eval run --agent api-claude

# Quick smoke test with mock tasks
openadapt eval mock --tasks 10

Why this matters: The waa benchmark (likely "Web/App Automation") provides standardized tasks with known success criteria. Running evaluation before production deployment catches failure modes early. The mock evaluation (--tasks 10) gives instant feedback on your setup without requiring full model weights.

Example 4: Viewing and Managing Captures

Inspect your recordings for quality assurance:

# List all recorded demonstrations
openadapt capture list

# Visualize a specific capture in your browser
openadapt capture view invoice-processing

The openadapt-viewer package renders an HTML visualization showing screenshots, action overlays, and accessibility tree snapshots synchronized in time. This is invaluable for debugging why a particular demonstration led to unexpected model behavior.

Example 5: Starting the Dashboard Server

For team collaboration and monitoring:

# Launch the web dashboard on port 8080
openadapt serve --port 8080

Navigate to http://localhost:8080 to access the dashboard. This centralizes capture management, training job monitoring, and evaluation results for team visibility.

Advanced Usage & Best Practices

Build a Demonstration Library, Not Single Tasks The power of OpenAdapt's retrieval path comes from demonstration diversity. Record the same task performed on different screen resolutions, with varying window arrangements, and across UI themes. This builds robustness that single-demonstration approaches lack.

Leverage the Abstraction Ladder Start with literal replay for critical tasks—exact mouse coordinates, precise timing. As you gain confidence, progress through template-based matching ("click the button labeled 'Submit'") to goal-level automation ("complete the payment flow"). The openadapt-ml training path supports this progression with different fine-tuning objectives.

Implement Multi-Model Consensus for Critical Operations For high-stakes automation, deploy openadapt-consilium to run multiple models in parallel. If Claude, GPT-4o, and Qwen3-VL all agree on an action, execute automatically. On disagreement, escalate to human review via openadapt-crier. This "swarm intelligence" pattern dramatically reduces error rates.

Privacy Scrub Before Any Sharing Even within your organization, run openadapt-privacy on all captures before adding them to shared demonstration libraries. The scrubbing runs locally—no data leaves your machine—removing names, emails, account numbers, and other sensitive patterns using both regex and ML-based detection.

Version Your Demonstrations Like Code Use openadapt-herald to auto-generate social media^{↗ Bright Coding Blog} and changelog content from your git history, but more importantly, tag demonstration datasets with semantic versions. "Invoice processing v2.1.0" is infinitely more maintainable than "that recording from March."

OpenAdapt vs. The Competition

The verdict: Traditional RPA tools demand you describe every interaction explicitly. Browser automations break when developers refactor CSS. Commercial AI agents offer convenience at the cost of control and transparency. OpenAdapt occupies the unique position of cutting-edge AI capability with full open-source freedom.

Frequently Asked Questions

Is OpenAdapt production-ready today? The v1.0+ modular architecture is actively developed with validated results on controlled benchmarks. The retrieval-conditioned prompting (Phase 2) is validated; demo-conditioned fine-tuning (Phase 3) is in progress. Start with supervised execution and progress to autonomy as your evaluation metrics support it.

What hardware do I need for local model training? The Qwen3-VL 2B model trains efficiently on consumer GPUs with 8GB+ VRAM. For larger models or faster training, cloud GPU instances work seamlessly. Inference with API-based models (Claude, GPT-4o) requires no local GPU.

Can OpenAdapt handle applications with heavy anti-automation? OpenAdapt operates at the OS input level and uses visual understanding, making it inherently harder to detect than DOM-manipulating tools. However, always respect Terms of Service and legal requirements for target applications.

How does demonstration privacy scrubbing work? openadapt-privacy combines pattern matching for known PII formats (SSNs, credit cards, emails) with transformer-based named entity recognition. All processing happens locally; no data is transmitted externally.

What happens when the UI changes significantly? This is where demo-conditioned inference shines. Unlike brittle selectors, OpenAdapt's visual understanding tolerates layout changes, theme updates, and responsive reflows. For major redesigns, record new demonstrations—the retrieval system will match to the most relevant examples automatically.

Can I contribute to the project? Absolutely. Join the Discord community, pick issues from any sub-package repository, and submit PRs. The modular architecture means you can contribute to openadapt-grounding without understanding openadapt-ml internals.

Is there a graphical interface, or only CLI? Both. openadapt-desktop provides a full GUI application, while openadapt-tray offers a lightweight system tray app for quick capture start/stop. The CLI remains the power-user interface for scripting and automation.

Conclusion: The Future of Automation Is Demonstration, Not Scripting

OpenAdapt represents a fundamental paradigm shift in how we automate computer interactions. Instead of painstakingly describing every click, scroll, and keystroke through brittle abstractions, we demonstrate and let intelligent models learn the underlying intent. The 100% first-action accuracy achieved through demo-conditioned prompting isn't just a number—it's proof that "show, don't tell" works for human-computer interaction automation.

The modular architecture gives you control. The open-source license gives you freedom. The multimodal foundation gives you robustness that rule-based systems simply cannot match.

Whether you're automating legacy healthcare systems, testing dynamic web applications, or building the next generation of AI agents, OpenAdapt provides the bridge between cutting-edge vision-language models and the interfaces humans actually use every day.

Stop writing scripts that break. Start demonstrating tasks that scale.

👉 Get started now: Clone the repository at github.com/OpenAdaptAI/OpenAdapt, install with pip install openadapt[all], and record your first demonstration in under five minutes. Join the Discord community to share what you build and help shape the future of generative process automation.

The robots aren't coming for your job—they're coming to do the boring parts of it. With OpenAdapt, you're the one teaching them how.