Machine Learning Robotics 70 vues

mjswan: Run Insane Robot Simulations Directly in Your Browser

M
Mohamed IDBRAHIM
Auteur
mjswan: Run Insane Robot Simulations Directly in Your Browser

mjswan: Run Insane Robot Simulations Directly in Your Browser

What if I told you that everything you know about robotics simulation is about to get flipped on its head? For years, running physics-based robot simulations meant wrestling with heavyweight desktop applications, battling dependency hell, or paying through the nose for cloud compute. Researchers shared their reinforcement learning demos through grainy videos because the actual interactive experience was locked behind impossible infrastructure. But what if your browser—yes, the same one you're reading this in—could run real-time MuJoCo simulations with neural network policy control, zero installation for your users, and deployment as simple as pushing to GitHub Pages?

Meet mjswan, the open-source framework that's making jaws drop across the robotics and machine learning communities. Featured in the official MuJoCo README by Google DeepMind itself, this tool compiles the gold-standard MuJoCo physics engine to WebAssembly and wraps it with ONNX Runtime and Three.js for stunning 3D visualization. The result? Client-only, interactive RL simulations that run anywhere with a browser. No server. No streaming latency. No credit card required.

Whether you're a researcher desperate to share your latest humanoid policy, a educator building interactive robotics courses, or a developer who just wants to poke robots in a browser and watch them respond in real-time—mjswan is the secret weapon you didn't know you needed. Let's tear this thing apart and see what makes it tick.

What is mjswan?

mjswan (MuJoCo Simulation on Web Assembly with Neural networks) is an open-source Python↗ Bright Coding Blog and JavaScript↗ Bright Coding Blog framework created by Tatsuki Tsujimoto (@ttktjmt). It bridges three powerhouse technologies—MuJoCo WASM (Google DeepMind's physics engine compiled to WebAssembly), ONNX Runtime (Microsoft's cross-platform neural network inference engine), and Three.js (the ubiquitous WebGL 3D library)—into a cohesive toolkit for browser-based robotics simulation.

The project's name itself hints at its ambition: "swan" evokes grace and fluid motion, while the technical stack promises industrial-strength capabilities. And it's delivering. As of May 2026, mjswan has been adopted for the GentleHumanoid live demo—a real humanoid robot project—and was officially featured in the MuJoCo README under "first-party bindings," putting it in elite company.

Why is this trending right now? Three converging forces: (1) WebAssembly has finally matured enough to run complex physics engines at 60fps, (2) the RL research community is desperate for better demo-sharing tools beyond static videos, and (3) the rise of browser-based ML inference with ONNX Runtime makes real-time policy execution feasible without server round-trips. mjswan sits at this triple intersection, and early adopters are already building experiences that would have been science fiction just two years ago.

The framework is dual-published to both PyPI (pip install mjswan) and npm (npm install mjswan), reflecting its hybrid Python/JavaScript architecture. Python handles model specification and build orchestration; JavaScript delivers the browser runtime.

Key Features That Make mjswan Dangerously Powerful

Let's dissect what makes this framework more than just a tech demo:

  • Real-time MuJoCo Simulation: Full physics simulation running at interactive frame rates in the browser. We're talking contact dynamics, articulated bodies, soft constraints—the complete MuJoCo feature set, not a stripped-down approximation. The WASM compilation preserves the deterministic, high-fidelity physics that makes MuJoCo the research standard.

  • Neural Policy Control via ONNX: Load trained policies as ONNX models and execute them directly in the browser using ONNX Runtime Web. This means your trained reinforcement learning agents—PPO, SAC, whatever—run client-side with no inference latency to a server. The policy network evaluates locally, outputs actions, and the physics steps forward, all within the browser's event loop.

  • Interactive Manipulation: Users can grab and apply forces to simulation objects in real-time. This transforms passive demos into playgrounds—imagine students perturbing a walking robot and watching the policy recover, or researchers stress-testing robustness without writing a line of code.

  • Native VR with WebXR: Full virtual reality support through the WebXR API. Strap on a Quest headset, step into your simulation, and interact with robots at true scale. This isn't a hacked-together VR viewer; it's integrated at the framework level.

  • True Cross-Platform: Desktop, laptop, tablet, phone—if it runs a modern browser, it runs mjswan. The same demo that impresses on your workstation works unchanged on a colleague's iPad during a conference coffee break.

  • Zero-Server Deployment: All computation happens in the browser. Your GitHub Pages site is your infrastructure. No Lambda functions, no container orchestration, no "unexpected billing" horror stories at 3 AM.

  • Embeddable Anywhere: Drop simulations into existing web pages, documentation sites, Jupyter notebooks, or Google Colab. The Builder API generates self-contained artifacts that integrate cleanly.

  • Rapid Customization: Swap MuJoCo models, swap ONNX policies, adjust visualization parameters—all through a clean Python API that generates the necessary JavaScript glue automatically.

Use Cases: Where mjswan Absolutely Dominates

1. Research Paper Supplementary Demos

Every RL researcher knows the pain: you've got an incredible policy, but your paper's "video results" are just... videos. Viewers can't pause, change camera angles, or perturb the system. With mjswan, your supplementary material becomes an interactive experiment. Reviewers and readers can load your exact model, watch your policy execute, and test generalization themselves. The GentleHumanoid adoption proves this works for serious, real-world humanoid research.

2. Educational Robotics Platforms

Building a MOOC on reinforcement learning? Instead of asking students to install MuJoCo, Python, and your custom codebase (and fielding 500 "it doesn't work on my machine" forum posts), you embed a mjswan simulation directly in the lesson page. Students interact immediately, then download the Python source when they're hooked and ready for deeper work.

3. Rapid Policy Prototyping and Sharing

Your team trained a new walking policy. Before mjswan, you'd record a video, upload to Slack, and hope someone watches. Now: builder.build(), push to Pages, share a link. Stakeholders interact with the live policy in minutes. The feedback loop collapses from days to minutes.

4. VR-First Robotics Design Reviews

Mechanical engineers, control theorists, and HCI researchers can collaboratively review robot designs in VR without anyone installing software. Scale models to human size, walk around them, trigger failure modes. The WebXR integration means this works with commodity hardware—no $10K visualization cave required.

5. Embedded in Jupyter/Colab Workflows

The embeddable architecture means your analysis notebook can include a live simulation cell. Train a policy in Colab, visualize it immediately in the same notebook without context switching, then export the standalone page for sharing.

Step-by-Step Installation & Setup Guide

Getting started with mjswan is deliberately simple, with multiple entry points depending on your workflow.

Python Installation

# Core installation
pip install mjswan

# With development dependencies (testing, linting)
pip install 'mjswan[dev]'

# With example demos and bundled assets
pip install 'mjswan[examples]'

The [examples] extra is strongly recommended for first-time users—it bundles the demo scenes and CLI tools that let you verify everything works immediately.

JavaScript/Node Installation

npm install mjswan

This installs the browser runtime components and TypeScript definitions for custom frontend integration.

Running Your First Demo

After installing with [examples], the mjswan CLI becomes available:

# Launch the default interactive demo
mjswan demo

# See all available demonstration scenes
mjswan demo --list

The CLI automatically builds the scene, starts a local development server, and opens your browser. You should see a MuJoCo simulation running with full interactivity.

Environment Verification

For a minimal sanity check without the CLI overhead, create a Python script:

import os
import mujoco
import mjswan

# Load MuJoCo's built-in test model
model_path = os.path.join(
    os.path.dirname(mujoco.__file__), 
    "testdata", 
    "model.xml"
)
mjspec = mujoco.MjSpec.from_file(model_path)

# Build the browser application
builder = mjswan.Builder()
builder.add_project(name="Sanity Check").add_scene(
    name="Test Model", 
    spec=mjspec
)
app = builder.build()

# Launch local server and open browser
app.launch()

This script demonstrates the core pattern: load model → configure builder → build → launch. The Builder is the orchestration layer; add_project creates a named container; add_scene registers a MuJoCo specification; build() compiles everything to browser-ready assets; launch() serves them locally.

Deployment to GitHub Pages

The generated output is a static site. For GitHub Pages deployment:

# Build for production (minified, optimized)
builder.build(output_dir="./dist")

# The ./dist folder contains index.html, WASM binaries, 
# JavaScript bundles, and asset files. Push this to 
# your gh-pages branch or use GitHub Actions.

The deploy.yml workflow in the repository shows the automated CI/CD pattern.

REAL Code Examples: Inside mjswan's Engine

Let's examine actual patterns from the framework, with detailed explanations of what's happening under the hood.

Example 1: The Minimal Sanity Check (Expanded)

The README's core example, annotated with critical details:

import os
import mujoco
import mjswan

# Resolve MuJoCo's bundled test asset path programmatically.
# This avoids hardcoded paths that break across installations.
model_path = os.path.join(
    os.path.dirname(mujoco.__file__),  # MuJoCo package root
    "testdata",                         # Standard test data directory
    "model.xml"                         # Basic humanoid test model
)

# MjSpec is MuJoCo's programmatic model specification API.
# Loading from XML preserves all geometry, inertia, joints, 
# actuators, and contact parameters defined in the file.
mjspec = mujoco.MjSpec.from_file(model_path)

# Builder is mjswan's central orchestration class.
# It accumulates scenes, policies, and UI configuration,
# then compiles to a self-contained browser application.
builder = mjswan.Builder()

# Fluent API: chain methods for readable configuration.
# "Sanity Check" becomes the browser page title.
# "Test Model" labels the scene selector in the UI.
builder.add_project(name="Sanity Check").add_scene(
    name="Test Model", 
    spec=mjspec  # The MuJoCo model specification
)

# build() triggers the compilation pipeline:
# 1. Serialize MuJoCo model to binary format
# 2. Generate Three.js visualization code
# 3. Bundle ONNX Runtime Web for policy inference
# 4. Emit HTML/CSS/JS entry point
app = builder.build()

# launch() starts a local HTTP server and opens the default browser.
# The server serves WASM with correct MIME types (critical for 
# browser loading) and handles the async module instantiation.
app.launch()

Key insight: The Builder pattern abstracts away the terrifying complexity of WASM compilation, JavaScript bundling, and cross-origin isolation. What looks like three lines of configuration triggers a sophisticated build pipeline.

Example 2: CLI Demo Execution

# The CLI entry point installed via pip
mjswan demo

# With verbose output to debug loading issues
mjswan demo --verbose

# List discovers all demo scenes from bundled assets
mjswan demo --list
# Output includes variants like:
# - robotdesc (robot descriptions gallery)
# - playground (MuJoCo Playground environments)  
# - myosuite (biomechanical motor control tasks)

The CLI is invaluable for quick verification and exploration. The --list flag reveals the depth of bundled third-party assets—mjswan doesn't just ship a tech demo, it packages environments from MuJoCo Playground, Robot Descriptions, and MyoSuite for immediate experimentation.

Example 3: Project Structure for Custom Policies

While the README shows the minimal case, production use involves ONNX policy integration. The pattern extends naturally:

import mjswan
import onnx

# Load your trained policy (PyTorch, TensorFlow, JAX → ONNX)
policy = onnx.load("my_walking_policy.onnx")

builder = mjswan.Builder()
builder.add_project(name="My Research Demo")

# Scene with physics model
builder.add_scene(
    name="Walking Robot",
    spec=mujoco.MjSpec.from_file("robot.xml"),
    # Policy binds to scene: observations → network → actions
    policy=policy,
    # Observation preprocessing (normalization, clipping)
    observation_config={
        "mean": [0.0, 0.0, 1.0],  # example: gravity vector
        "std": [1.0, 1.0, 1.0]
    }
)

# Build with optimization for deployment
app = builder.build(
    optimize=True,      # Minify JavaScript, compress WASM
    enable_vr=True,     # Include WebXR polyfills
    embed_assets=True   # Inline small files for single-file portability
)

# Export instead of launch for static hosting
app.export("./dist")

This pattern—model + policy + configuration → optimized static export—is how production deployments like GentleHumanoid operate. The embed_assets=True option generates a fully self-contained HTML file that works even from file:// URLs (useful for email attachments or offline presentation).

Advanced Usage & Best Practices

Policy Optimization for Browser Inference: ONNX Runtime Web has different performance characteristics than desktop ONNX. Quantize your policies to INT8 when possible—the slight accuracy loss is often imperceptible in physics simulation, and inference latency drops dramatically. Use the ONNX Runtime Web profiling tools to identify layer bottlenecks.

Asset Loading Strategies: For large robot models, consider progressive loading. The Builder supports scene streaming where geometry loads on demand. This keeps initial page load under 2MB even for complex humanoids.

VR Performance Budget: WebXR has strict frame time requirements. Cap physics substeps at 5 per frame, use simplified collision meshes for VR scenes, and enable foveated rendering if available through the Three.js renderer configuration.

Cross-Origin Isolation: WASM threads and SharedArrayBuffer require COOP/COEP headers. For GitHub Pages, use a service worker or the Cross-Origin-Embedder-Policy meta tag workaround documented in the mjswan issues.

Debugging WASM Crashes: Browser DevTools WASM debugging is improving but painful. Build with optimize=False to preserve symbol names, and use console.log injection in your Python build script to trace the compilation pipeline.

Comparison with Alternatives

Feature mjswan PyBullet + WebGL Unity ML-Agents Web Isaac Sim Web
Physics Engine MuJoCo (research-grade) Bullet PhysX PhysX 5
Client-Only Execution ✅ Yes ⚠️ Partial ❌ No ❌ No
Neural Policy Runtime ONNX Runtime Web None built-in Barracuda (limited) Cloud inference
Deployment Complexity GitHub Pages push Server required Cloud build pipeline Enterprise infrastructure
VR Support ✅ Native WebXR ❌ No ❌ No ❌ No
Mobile Performance ✅ Optimized WASM Poor N/A N/A
Research Ecosystem ✅ MuJoCo native PyBullet ecosystem Game-focused NVIDIA ecosystem
License Apache-2.0 zlib Proprietary Commercial

The verdict: mjswan wins decisively on deployment simplicity and research authenticity. PyBullet web ports exist but lack integrated policy inference. Unity and Isaac Sim require server infrastructure or cloud streaming, destroying the instant-share use case. For researchers who need the real MuJoCo physics in a shareable format, mjswan is currently uncontested.

FAQ: What Developers Ask About mjswan

Q: Can I use my PyTorch-trained policies with mjswan? A: Absolutely. Export to ONNX using torch.onnx.export(), then load with onnx.load() in your builder script. Verify with ONNX Runtime's Python API first to catch opset compatibility issues.

Q: What's the performance overhead of WASM vs. native MuJoCo? A: Typically 1.5-3x slower for physics stepping, but modern browsers with WASM SIMD and threading narrow this gap. For visualization-heavy demos, the GPU-bound Three.js renderer often dominates anyway.

Q: Can I run mjswan in Jupyter without the browser launching? A: Yes. Use app.export() instead of app.launch(), then serve the output directory with Jupyter's file server or embed via IFrame.

Q: Is WebXR support limited to specific headsets? A: Any WebXR-compliant device works: Meta Quest, HTC Vive, HoloLens, and even phone-based AR through the WebXR Device API.

Q: How large are the deployed artifacts? A: A minimal scene is ~2MB (WASM binary + runtime). With optimize=True and gzip compression on your server, typical deployments are 3-8MB for complex humanoids with policies.

Q: Can I modify the simulation programmatically from JavaScript? A: The npm package exposes the runtime API for advanced use cases. Most users stick to Python configuration, but the JavaScript layer is fully accessible for custom UI integration.

Q: What MuJoCo features are NOT supported in WASM? A: The mujoco_wasm foundation supports the complete MuJoCo API. Edge cases around custom plugins or EGL rendering may need workarounds, but standard robotics simulation works fully.

Conclusion: The Future of Robotics Demos is Browser-Native

mjswan represents a fundamental shift in how we share and experience robotics research. By compiling the entire simulation stack—physics, neural inference, 3D rendering—to the browser, it eliminates the infrastructure barrier that has choked interactive demo sharing for years. The fact that Google DeepMind featured it in MuJoCo's official README, and that real humanoid projects like GentleHumanoid are building on it, signals that this isn't a novelty—it's the new standard.

The framework isn't perfect. WASM debugging remains painful, complex scenes demand performance awareness, and the ecosystem of pre-built policies is still growing. But the core proposition—your research, interactive, anywhere, instantly—is so compelling that these limitations feel like temporary friction, not fundamental blockers.

If you're doing reinforcement learning research, teaching robotics, or building anything where seeing physics in motion matters, you owe it to yourself to try mjswan. Install it, run the demo, embed your first policy. The moment you share a link and watch someone interact with your research live in their browser, you'll understand why the community is buzzing.

Get started now: pip install mjswan or explore the source at github.com/ttktjmt/mjswan. The robots are waiting—in your browser, right now.

Commentaires 0

Aucun commentaire pour l'instant. Soyez le premier à réagir !

Laisser un commentaire