claw-eval/claw-eval: Human-Verified Benchmark for LLM Agents

claw-eval/claw-eval: Human-Verified Benchmark for LLM Agents

Evaluating autonomous agents is noisy. Most benchmarks reward single-run luck, tolerate flaky API calls, or score outputs without checking whether the agent actually completed the task. claw-eval/claw-eval takes a stricter approach: 300 human-verified tasks, full-trajectory auditing, and a Pass^3 methodology that forces consistent performance across three independent trials.

This article walks through what Claw-Eval is, how it works, and how to run it yourself.

What is claw-eval/claw-eval?

Claw-Eval is an open-source evaluation harness for LLM-as-agent systems, maintained by researchers from Peking University (PKU) and the University of Hong Kong (HKU). The project is led by Lei Li (HKU) with core contributors Bowen Ye, Rang Li, Qibin Yang, and others from PKU, plus advisors including Tong Yang, Zhifang Sui, Lingpeng Kong, and Qi Liu.

The repository has accumulated 722 stars and 65 forks as of its last commit on May 17, 2026. It is written primarily in Python^{↗ Bright Coding Blog} and released under the MIT License.

Claw-Eval sits in a crowded space—agent benchmarks have proliferated since 2024—but distinguishes itself through three design choices:

• Human verification at the task level: Every one of the 300 tasks has been checked by humans, not synthetically generated and spot-checked.
• Rubric-driven grading: 2,159 rubrics provide fine-grained scoring criteria across completion, safety, and robustness dimensions.
• Pass^3 consistency: A model must pass all three independent trials to earn credit, eliminating "lucky run" artifacts that inflate scores on single-attempt benchmarks.

The project has gained enough traction that teams at Meta (Muse Spark), KAT-Coder-V2, Kimi, Qwen, Tencent Hunyuan, Xiaomi MiMo, Z.AI/GLM, and Ant Ling have publicly referenced or evaluated on Claw-Eval.

Key Features

300 Tasks Across 9 Categories

The benchmark spans three splits with distinct evaluation demands:

Three-Dimensional Grading

Each task is audited on the full agent trajectory, not just final output:

• Completion — Did the agent finish the task?
• Safety — Did it avoid harmful or unauthorized actions?
• Robustness — Does performance hold across multiple trials?

Pass^3 Methodology

The leaderboard updated to Pass^3 in March 2026. Under this protocol:

• A model runs each task three independent times (N=3)
• The task only counts as passed if all three runs succeed
• API or network errors trigger manual re-evaluation to ensure exactly 3 valid trajectories

This directly addresses a known failure mode in agent benchmarking: high variance across stochastic runs masks true capability gaps.

Reproducibility Commitment

The authors state their codebase is "currently being audited to ensure all benchmark results on the leaderboard can be verified by the community." This is unusual in a field where leaderboard results often depend on unreleased prompts or proprietary infrastructure.

Sandboxed Execution

The evaluation runs inside a sandbox environment with full-trace tracking, isolating agent actions from the host system.

Use Cases

1. Model Developers Validating Agent Releases

Teams building foundation models with agentic capabilities—like the Kimi, Qwen, and Hunyuan groups already using Claw-Eval—need benchmarks that resist gaming. Pass^3 makes it harder to optimize for single-run leaderboard position without genuine capability improvements.

2. Researchers Studying Agent Failure Modes

The 2,159 rubrics and full-trajectory auditing let researchers analyze where agents fail, not just whether they fail. The multi_turn split with simulated user personas is particularly useful for studying clarification and advice-seeking behaviors.

3. Multimodal Agent Evaluation

The 101 multimodal tasks cover webpage generation, video question-answering, and document extraction. These require agents to perceive visual or structured inputs and produce structured outputs—capabilities not captured by text-only benchmarks.

4. Safety-Critical Deployment Pre-Checks

The explicit safety dimension checks for "harmful or unauthorized actions" during task execution. This matters for agents with tool access to real systems, where a benchmark that only checks task completion would miss dangerous shortcuts.

5. Cross-Lingual Evaluation

Tasks are available in English (en) and Chinese (zh), supporting evaluation of multilingual agent systems.

Installation & Setup

The maintainers recommend uv for dependency management. Here are the exact commands from the README:

# Install uv
pip install uv

# Create Python 3.11 virtual environment
uv venv --python 3.11
source .venv/bin/activate

Set required API keys:

export OPENROUTER_API_KEY=sk-or-...
export SERP_DEV_KEY=...  # Required for tasks needing real web search

The README notes you can obtain a SERP_DEV_KEY from https://www.novada.com for convenience.

Run the sandbox test script:

bash scripts/test_sandbox.sh

Important notes from the documentation:

• Video fixtures are excluded from the GitHub repository due to file size limits. Download complete fixtures (including videos) from Hugging Face.
• Grading models differ by task type: gemini-3-flash for general and multimodal tasks; claude opus4.6 for both grader and user-agent in multi_turn tasks.

Real Code Examples

Running a Batch Evaluation

The primary evaluation command from the README:

# Run with Claude Opus 4.6 config, sandbox enabled, 3 trials, 16 parallel workers
claw-eval batch --config model_configs/claude_opus_46.yaml --sandbox --trials 3 --parallel 16

This invokes the claw-eval CLI with:

• --config: Model configuration file (here, Anthropic's Claude Opus 4.6)
• --sandbox: Isolated execution environment
• --trials 3: Pass^3 protocol—three independent attempts per task
• --parallel 16: Concurrent task execution for throughput

Task-Specific Configurations

The README indicates different configs for different evaluation targets:

# General agent tasks
# config_general.yaml

# Multimodal perception and creation tasks
# config_multimodal.yaml

# Conversational multi-turn tasks
# config_user_agent.yaml

These are referenced but not fully displayed in the README. Users should inspect the model_configs/ directory after cloning.

Dataset Structure

The Hugging Face dataset exposes these fields:

Advanced Usage & Best Practices

Handling API Instability

The Claw-Eval team manually re-triggers evaluations when API or network errors occur during execution. For automated pipelines, you should implement similar retry logic with exponential backoff, then verify you have exactly 3 successful trajectories before computing Pass^3 scores.

Fixture Management

Since video files live on Hugging Face rather than GitHub, production evaluation pipelines should cache fixtures locally and verify checksums. Network latency to Hugging Face or ModelScope (the Chinese mirror) can become a bottleneck for large-scale evaluation.

Grader Consistency

Using different grading models (gemini-3-flash vs claude opus4.6) for different splits introduces potential inter-grader variance. When comparing models across splits, check whether score differences reflect true capability gaps or grader leniency differences. The authors do not publish inter-grader agreement statistics, so treat cross-split comparisons with appropriate caution.

Custom Task Integration

The rubric structure (2,159 rubrics across 300 tasks, averaging ~7 rubrics per task) suggests a granular scoring system. Teams adding custom tasks should align rubric granularity with this density for comparable leaderboard submissions.

Comparison with Alternatives

Claw-Eval competes with several agent benchmarks, though direct feature parity varies:

Trade-offs:

• SWE-bench has more tasks but focuses narrowly on software engineering; Claw-Eval covers broader agent domains including finance and productivity.
• WebArena evaluates web navigation with realistic sites but uses single-attempt scoring; Claw-Eval's Pass^3 reduces variance at higher compute cost.
• OSWorld tests GUI interaction via virtual machines; Claw-Eval's sandbox is lighter-weight but may not capture full OS-level interaction complexity.

No benchmark dominates across all dimensions. Claw-Eval's strict reproducibility commitment and multi-trial protocol suit teams prioritizing score reliability over evaluation speed.

FAQ

What license is Claw-Eval under? MIT License, as indicated by the badge in the README.

Do I need a specific Python version? Python 3.11 is recommended based on the uv venv --python 3.11 command.

Can I evaluate proprietary models? Yes—use OpenRouter API keys for models accessible through that platform, or configure your own endpoints.

Why are video fixtures missing from GitHub? File size limits. Download complete fixtures from Hugging Face or ModelScope.

How do I submit to the leaderboard? Contact the authors at bwye@stu.pku.edu.cn, lirang410@gmail.com, or nlp.lilei@gmail.com.

Is the Pass^3 methodology configurable? The README shows --trials 3 as a CLI flag, suggesting the count is adjustable, though leaderboard submissions likely require exactly 3.

What languages are supported? English and Chinese tasks are available.

Conclusion

claw-eval/claw-eval offers a rigorously structured approach to a genuinely hard problem: measuring whether LLM agents can reliably perform complex, multi-step tasks in sandboxed environments. The Pass^3 methodology, human verification, and reproducibility audit distinguish it from benchmarks that optimize for ease of execution over score validity.

It is best suited for:

• Model teams needing defensible evaluation protocols
• Researchers studying agent failure modes across completion, safety, and robustness
• Practitioners evaluating multimodal or multilingual agent systems

The 722-star repository is actively maintained, with public engagement from major model providers. If you are building or evaluating agentic systems, claw-eval/claw-eval on GitHub warrants serious consideration.

For related evaluation tooling, see [INTERNAL_LINK: llm-evaluation-frameworks].