Stop Wasting GPU Cycles! Run Litmus as Your Autonomous ML Research Lab

Stop Wasting GPU Cycles! Run Litmus as Your Autonomous ML Research Lab

What if your GPU could discover breakthroughs while you sleep? Not just train models—actually think, experiment, and learn from its own failures?

Here's the brutal truth most ML engineers won't admit: your expensive GPU sits idle 70% of the time. Between meetings, code reviews, and debugging, that $4,000 A100 is doing nothing but collecting dust. Meanwhile, your research backlog grows. Promising architecture ideas go untested. Hyperparameter combinations that could unlock state-of-the-art results remain unexplored.

What if I told you there's a way to transform those wasted hours into a self-directing research organization?

Enter Litmus—the OpenClaw skill that turns your always-on machine into a parallel autonomous ML research lab. Multiple agents running experiments overnight. A Director steering them every two hours. A Synthesizer distilling knowledge into reusable skills. And every morning at 08:00, a research narrative lands in your chat with what your agents discovered while you were dreaming.

This isn't science fiction. This is Litmus, built on Andrej Karpathy's autoresearch foundation but reimagined for serious production use. And in this guide, I'll show you exactly how to deploy it.

What is Litmus?

Litmus is an OpenClaw skill created by Kuberwastaken that transforms a single GPU machine into a self-directing machine learning research organization. Unlike traditional experiment runners that execute predefined scripts, Litmus operates as a multi-agent system where native subagents autonomously design, execute, and learn from experiments—each operating on its own git branch with full version control.

The project builds directly on Karpathy's autoresearch, but addresses its critical limitations. Where autoresearch runs a single agent with per-run logs and no knowledge accumulation, Litmus deploys 2–8 parallel workers with full git-based experiment history, cross-agent learning through shared discovery files, and a nightly knowledge distillation pipeline that prevents your agents from rediscovering the same dead ends.

Why it's trending now: The ML research community is hitting a wall with manual experimentation. With models growing exponentially in complexity, the search space for architecture and optimization decisions has become intractable for human-only exploration. Litmus represents a shift toward compute-driven research automation—using your existing hardware to systematically explore hypotheses that would take weeks of human effort. The integration with OpenClaw's sessions_spawn architecture means zero external process management, no PID file headaches, and seamless conversational setup.

Version 1.1.1 is available under MIT-0 license, with active deployment automation and a growing community on ClawHub.

Key Features That Separate Litmus From Everything Else

Parallel Agent Architecture with Git-Native Versioning

Litmus spawns 2–8 worker subagents through OpenClaw's native sessions_spawn mechanism. Each worker operates on its own git branch within ~/.litmus/repo/, creating a complete experiment tree you can browse, cherry-pick, and analyze. This isn't logging—this is full version control for research.

The Director: Active Steering Every 2 Hours

A Director cron job reviews all shared attempts, computes improvement rates, and triggers Compass Reset when agents stagnate (≥6 experiments without improvement). It reads skills gaps and git history to steer workers toward unexplored combinations. No more agents circling the same hyperparameter neighborhood.

The Synthesizer: Nightly Knowledge Distillation

At 04:00 daily, the Synthesizer reads all notes and attempts from the past 24 hours, then writes validated techniques into shared/skills/ and updates the research agenda. This prevents rediscovery—once an agent finds a useful technique, all future agents build upon it.

Circadian Rhythm for Creative Thinking

The litmus-leisure mode (03:00–06:00 by default) shifts agents from execution to speculative thinking. They scan arxiv, analyze contradictions in recent papers, and write moonshot hypotheses to shared/notes/moonshots/. Your agents literally dream up new ideas overnight.

Two-Phase Budget with Dead-End Pruning

Every experiment starts with a 90-second quick check before committing to the full 300-second run. Agents abandon unpromising directions early, preserving compute for high-potential explorations.

Structured JSON Attempt Records

Each experiment generates a machine-readable record in shared/attempts/<hash>.json containing agent identifier, validation metrics, status, title, commit hash, and parent experiment. This enables programmatic analysis and leaderboard generation.

Morning Digest Delivery

At 08:00, litmus-digest compiles a research narrative from all overnight activity and delivers it to your chat. Wake up to actual discoveries, not just training logs.

Use Cases Where Litmus Absolutely Dominates

1. Architecture Search for Transformer Variants

Testing different depth/aspect ratio combinations, head dimensions, and window patterns (SLSL/LSLS/etc.) manually is excruciating. Litmus's architecture template deploys multiple agents to explore this space systematically, with the Synthesizer identifying which structural properties correlate with improved validation BPB. One user reported discovering a 12% perplexity improvement from an unusual depth-width combination their agents found on night three.

2. Optimizer Hyperparameter Landscape Mapping

Per-matrix learning rates, schedule shapes, Muon vs AdamW comparisons—these require hundreds of experiments. Litmus agents run these in parallel with automatic budget allocation, while the Director prevents overlap and the Synthesizer builds a reusable optimizer configuration library validated across multiple model sizes.

3. Regularization Technique Discovery

Softcap values, gradient clipping thresholds, weight decay schedules, residual scaling factors—interact in complex ways. Litmus's general template treats this as a combinatorial exploration problem, with the Compass Reset mechanism ensuring agents don't get trapped in local optima of the hyperparameter space.

4. Paper-Grounded Novel Hypothesis Generation

The leisure mode's arxiv scan → contradiction analysis → moonshot hypothesis pipeline means your agents can read today's papers and design experiments to test conflicting claims overnight. Imagine waking up to an experiment queue targeting a specific disagreement between two NeurIPS papers.

5. Continuous Background Research on Shared Infrastructure

For teams with always-on development machines, Litmus turns idle periods into collective knowledge accumulation. Multiple researchers' agents share the skills library, building organizational memory that persists across projects and personnel changes.

Step-by-Step Installation & Setup Guide

Prerequisites

Before installation, verify your environment meets these requirements:

• OS: Linux or macOS (Windows not supported)
• GPU: NVIDIA with CUDA support
• uv: Modern Python^{↗ Bright Coding Blog} package manager
• git: For experiment version control and worktrees
• python3: For JSON attempt records and leaderboard scripts

Install uv if you haven't already:

curl -LsSf https://astral.sh/uv/install.sh | sh

One-Command Installation

The beauty of Litmus is its conversational setup. Simply tell your OpenClaw agent:

"Install https://clawhub.ai/kuberwastaken/litmus and set it up for my machine"

Your agent will:

1. Check your GPU and determine optimal agent count
2. Pitch a full schedule with timing presets (Standard, Night owl, or Early bird)
3. Spawn worker subagents on dedicated git branches
4. Register all cron jobs automatically

All in one continuous conversation. No manual cron editing. No shell script wrangling.

Manual Cron Configuration (Optional)

For custom scheduling, pass parameters directly to setup-cron.sh:

# Example: Night owl schedule with custom director hours
bash ~/.litmus/scripts/setup-cron.sh \
  --leisure-start 01:00 \
  --synthesizer-time 02:00 \
  --dawn-time 04:00 \
  --digest-time 07:00 \
  --director-hours "22:00-06:00" \
  --watchdog-minutes 30

Configuration File

All settings persist in ~/.litmus/config.json. Key defaults:

Verifying Installation

Check your agents' status immediately after setup:

# Full system status with per-agent metrics
bash ~/.litmus/scripts/status.sh

# Top 10 experiments across all agents
bash ~/.litmus/scripts/results.sh --top 10

# Visualize the complete experiment tree
git -C ~/.litmus/repo log --all --oneline --graph

REAL Code Examples from the Repository

Example 1: Inspecting the Complete Experiment Tree

The git-native architecture is Litmus's secret weapon. Here's how to browse your entire research history:

# View all experiment branches with their relationships
git -C ~/.litmus/repo log --all --oneline --graph

This simple command reveals the full topology of your agents' exploration. Unlike traditional experiment trackers that show flat lists, you see branching decisions, merge points where agents incorporated each other's findings, and abandoned directions pruned by the two-phase budget. The --graph visualization makes it immediately obvious which agents found promising subspaces and where the Director triggered Compass Reset to escape stagnation.

For deeper inspection of any specific experiment:

# Show full diff and metadata for a commit
git -C ~/.litmus/repo show <commit-hash>

# Read the structured JSON record
cat ~/.litmus/shared/attempts/<hash>.json

The JSON record contains machine-parseable results that feed the leaderboard and Synthesizer. This dual representation—human-readable git history plus structured data—enables both intuitive exploration and automated analysis.

Example 2: Runtime Agent Steering Without Restart

Litmus enables mid-flight course correction, a capability absent from most experiment frameworks:

# Steer a specific worker to combine findings from another agent
subagents action:"steer" target:"litmus-worker-arch-1" \
  message:"Checkout opt-2's best commit and combine their LR with DEPTH=10."

This is extraordinary. The subagents command with action:"steer" sends an instruction to a running worker without interrupting its session. The target agent can incorporate cross-branch insights in real-time. In this example, arch-1 is directed to examine opt-2's best configuration and hybridize its learning rate schedule with its own depth parameter.

The message: payload is interpreted by the agent's context, enabling natural language direction of complex experimental strategy. This human-in-the-loop override preserves autonomy while allowing expert intervention when you spot opportunities your agents haven't.

Emergency stop capability uses the same interface:

# Halt all agents immediately
subagents action:"kill" target:"all"

Or simply tell your OpenClaw agent conversationally: "Stop all Litmus agents". The natural language interface masks sophisticated process management—no PID hunting, no kill -9 roulette.

Example 3: Leaderboard and Results Analysis

Litmus generates actionable intelligence from raw experiment data:

# Global leaderboard across all agents, top 10 by validation BPB
bash ~/.litmus/scripts/results.sh --top 10

# Single-agent deep dive for debugging or publication
bash ~/.litmus/scripts/results.sh --agent arch-1

These scripts parse the shared/attempts/ JSON records to compute rankings, identify statistical trends, and flag anomalies. The --agent filter is crucial for attribution—understanding which agent strategies produce consistent improvements versus lucky outliers.

The status script provides operational intelligence:

# Per-agent experiment counts, best val_bpb, stagnation flags, git tree visualization
bash ~/.litmus/scripts/status.sh

This outputs critical health metrics: experiment velocity (are agents actually running?), best achieved metric per agent, stagnation flags (has Compass Reset triggered?), and a compact git tree representation. Use this for morning triage—identify which agents need intervention before the Director's next cycle.

Example 4: Conversational Management

The simplest interface is often the most powerful:

# Natural language status check
"How are my Litmus agents doing?"

# Natural language shutdown
"Stop all Litmus agents"

These aren't shell aliases—they're interpreted by your OpenClaw agent which maps intent to the appropriate subagents API calls. This abstraction means you manage a complex distributed system as casually as checking the weather.

Advanced Usage & Best Practices

Template Selection Strategy

Start with architecture for model structure questions, optimizer for training dynamics, and general for open-ended exploration. Combine templates by running multiple agent groups with different focuses—the Director will cross-pollinate findings automatically.

Custom Goal Injection

Set custom_goal in config for targeted campaigns. Example: "Find configurations that maintain < 100M parameters while exceeding baseline perplexity by 15%". The Synthesizer weights attempts toward this objective.

Leisure Intensity Tuning

Standard leisure mode (3 arxiv searches, 5 papers, 5 moonshots) balances thoroughness with compute preservation. For literature-heavy research phases, increase intensity. For pure experimentation phases, reduce or disable leisure to maximize experiment budget.

Skills Library Curation

Periodically review shared/skills/ and prune obsolete techniques. The Synthesizer is conservative about validation—human curation prevents accumulation of context-specific skills that don't generalize.

Disk Management

The litmus-archive job runs every 3 days, but monitor ~/.litmus/shared/attempts/ growth. Large-scale campaigns generate substantial JSON. Archive old attempts to cold storage after Synthesizer processing.

Escape Mode Awareness

The watchdog triggers escape mode on zero improvements across all agents. This is a system-level signal that your research direction is exhausted. Don't override blindly—investigate whether your goal is too constrained or the search space genuinely depleted.

Comparison with Alternatives

The decisive advantage: Litmus is the only system that accumulates and transfers knowledge across experiments and agents. Every other tool treats each run as independent. Litmus builds a compounding research asset.

FAQ

Q: How much GPU memory do I need for Litmus? A: Agent count scales with available VRAM. A single 24GB GPU typically runs 2–3 agents comfortably. The onboarding conversation auto-detects and recommends configuration.

Q: Can I run Litmus without OpenClaw? A: No—Litmus is architected as an OpenClaw skill using native sessions_spawn subagents. The tight integration enables conversational management and eliminates external process overhead.

Q: What happens if an agent crashes overnight? A: The litmus-watchdog cron runs every 30 minutes with liveness checks. Failed agents are flagged in status output, and the Director redistributes their queue on next cycle.

Q: How do I prevent agents from wasting compute on bad ideas? A: The two-phase budget (90s quick check before 300s full run) eliminates most dead ends. Additionally, the skills library prevents re-testing known failures.

Q: Can I export experiments to publishable format? A: Yes—every experiment is a git commit with full reproducibility. The JSON attempt records contain all hyperparameters and results. Combine with git show for complete methodological transparency.

Q: Is my data shared with external services? A: No. All computation runs locally. The arxiv scan in leisure mode uses public API, but no experiment data leaves your machine.

Q: How do I update Litmus when new versions release? A: Ask your OpenClaw agent: "Update my Litmus installation". The skill handles migration of existing experiments and config.

Conclusion

Litmus represents a fundamental shift in how we approach ML research. Not as a sequence of manually designed experiments, but as an ongoing autonomous process that compounds knowledge, explores systematically, and delivers actionable intelligence every morning.

The comparison is stark: traditional experimentation is artisanal—beautiful, slow, and limited by human attention. Litmus is industrial—parallel, self-improving, and bounded only by your compute budget.

I've seen too many brilliant researchers burn months on hyperparameter grids that agents could explore in nights. Too many papers citing "future work" on combinations that sit untested because no human had time. Litmus doesn't eliminate the researcher—it amplifies them, handling systematic exploration while you focus on the creative leaps only humans can make.

Your GPU is waiting. Those idle hours are research debt accumulating. Install Litmus tonight, check your chat at 08:00 tomorrow, and discover what your machines can learn without you.

→ Install Litmus on your OpenClaw instance now

→ Explore the ClawHub skill page

→ Read the full documentation