Stop Shipping Broken Agent Skills! Use Skillgrade Instead

Stop Shipping Broken Agent Skills! Use Skillgrade Instead

Your AI agent just failed in production. Again. It was supposed to fix linting errors, but it deleted the entire src directory. Or maybe it was supposed to generate a React^{↗ Bright Coding Blog} component, and it produced 400 lines of incomprehensible spaghetti code that doesn't even compile. You thought you tested it. You swore it worked in your demo. But here's the brutal truth: you never actually tested it. You eyeballed it. You ran it once on your machine. You called it "good enough." And now your users are paying the price.

This is the dirty secret of the AI agent revolution. Everyone's building skills. Everyone's shipping "agent-powered" features. Almost nobody is systematically evaluating whether those skills actually work. We're in the Wild West of agent development—cowboys coding by campfire light, hoping their creations don't burn down the ranch. But what if there was a better way? What if you could apply the same rigorous testing discipline that transformed software engineering to the chaotic frontier of AI agents?

Enter Skillgrade, the open-source evaluation framework created by Minko Gechev. Think of it as "unit tests for your agent skills"—a systematic, repeatable, automatable way to verify that your AI agents correctly discover, understand, and execute the skills you've built for them. No more hoping. No more praying. Just hard data on whether your agents actually deliver. In this deep dive, I'll show you why Skillgrade is about to become the essential tool in every serious agent developer's arsenal—and how you can start using it in the next 15 minutes.

What is Skillgrade?

Skillgrade is a command-line evaluation framework designed specifically for testing Agent Skills—the structured capabilities that AI coding agents use to perform tasks in your codebase. Created by Minko Gechev, a well-known figure in the Angular and web development^{↗ Bright Coding Blog} community, Skillgrade addresses a critical gap in the agent development lifecycle: the complete absence of systematic testing.

The project emerged from a clear-eyed recognition of where the industry stands. As Gechev's own references acknowledge, Skillgrade draws inspiration from academic work like SkillsBench and Anthropic's research on demystifying evals for AI agents. This isn't hobbyist tinkering—it's a production-ready response to a production-ready problem.

Why it's trending now: We're witnessing an explosion of AI agent frameworks—Claude Code, Codex, Gemini CLI, OpenCode, and countless others. But with this proliferation comes a crisis of quality. How do you know your skill works across different agents? How do you catch regressions when you update your skill's instructions? How do you prove to stakeholders that your "AI-powered" feature actually functions? Skillgrade answers all three questions with a single, elegant tool.

The framework supports multiple AI agents (Gemini, Claude, Codex, OpenCode, ACP-compatible agents), multiple execution providers (Docker^{↗ Bright Coding Blog} for isolation, local for speed), and two distinct grading strategies—deterministic checks for objective correctness, and LLM rubrics for qualitative evaluation. This flexibility makes it equally valuable for individual developers shipping their first skill and enterprise teams managing dozens of agent capabilities across complex codebases.

Key Features That Make Skillgrade Essential

Multi-Agent Support with Auto-Detection. Skillgrade doesn't lock you into a single AI provider. Set GEMINI_API_KEY, ANTHROPIC_API_KEY, or OPENAI_API_KEY, and the framework automatically routes to the correct agent. Override with --agent=claude when you need to test cross-agent compatibility. This matters because your users won't all use the same agent—and your skill should work everywhere.

Dual Grading Architecture. The framework's genius lies in combining two evaluation paradigms. Deterministic graders run scripts that produce machine-parseable JSON scores—perfect for "did the file get created?" or "does the output match?" LLM rubric graders leverage language models themselves to evaluate qualitative criteria like workflow compliance, efficiency, and code quality. Weight them together for nuanced scoring that captures both functional correctness and execution excellence.

Docker-First Isolation, Local Speed. By default, Skillgrade runs evaluations in Docker containers using configurable base images. This protects your development machine from rogue agent actions—crucial when testing skills that modify files. In CI environments, switch to --provider=local for faster execution without the container overhead. The same test suite runs identically in both modes.

AI-Powered Initialization. Run skillgrade init and the framework generates evaluation tasks using your preferred AI model. No staring at blank YAML files wondering what to test. The generated eval.yaml serves as a sophisticated starting point that you refine, not a rigid template you fight against.

CI-Native Design. Built-in --ci mode exits non-zero when pass rates fall below configurable thresholds. Reports persist to disk for artifact collection. The web UI (skillgrade preview browser) spins up on localhost:3847 for interactive debugging. This isn't a side project—it's engineered for production pipelines.

Workspace Orchestration. The workspace configuration lets you precisely control what files enter the evaluation environment—copy source files, install binaries, set permissions. Your test fixtures mirror production conditions without polluting your repository.

Real-World Use Cases Where Skillgrade Shines

Validating Developer Tooling Skills. Imagine you've built a skill that helps agents fix ESLint violations across a codebase. Without Skillgrade, you manually test a few files, declare victory, and ship. With Skillgrade, you define tasks that present deliberately broken files, measure whether the agent produces correct output, and verify the fix with both deterministic parsing and qualitative rubric evaluation. Regression testing becomes automatic—change your skill's SKILL.md instructions, run the suite, know immediately if you broke something.

Cross-Agent Compatibility Testing. Your enterprise standardizes on Claude internally, but customers use Gemini and Codex. Skillgrade lets you run identical evaluations across all three agents with a single command change. Discover that Gemini misinterprets your file path instructions, or that Codex requires more explicit output naming. Fix once, validate everywhere.

Qualitative Workflow Evaluation. Not everything reduces to pass/fail. Does the agent follow your mandated "check → fix → verify" workflow, or does it blindly overwrite files? Does it ask clarifying questions when requirements are ambiguous? LLM rubric graders capture these behavioral dimensions, scoring agents on process quality, not just output correctness.

Continuous Integration for Agent Infrastructure. As your skill library grows, Skillgrade becomes your safety net. Integrate into GitHub Actions (example provided in the README) to block merges that degrade agent performance. Set thresholds at 80% for development, 95% for production releases. Treat agent skills with the same rigor as any other code dependency.

Benchmarking Agent Evolution. AI models improve constantly—sometimes unexpectedly. A skill that worked perfectly with Claude 3.7 might fail with 4.0. Skillgrade's preset flags (--smoke for 5 quick trials, --reliable for 15, --regression for 30) let you tune evaluation depth to your confidence needs. Track performance over time, catch model regressions before they reach users.

Step-by-Step Installation & Setup Guide

Getting started with Skillgrade takes under ten minutes if you have the prerequisites ready.

Prerequisites

• Node.js 20+ (the framework targets modern Node; Docker base images use node:20-slim)
• Docker (for isolated evaluation environments; optional for --provider=local)
• An API key for at least one supported AI provider: Google (Gemini), Anthropic (Claude), or OpenAI (Codex)

Global Installation

Install Skillgrade globally via npm for command-line access anywhere:

npm i -g skillgrade

This provides the skillgrade CLI with all subcommands: init, preview, and execution flags.

Project Initialization

Navigate to your skill directory—this must contain a SKILL.md file describing your agent capability:

cd my-skill/
GEMINI_API_KEY=your-key skillgrade init

The init command is where Skillgrade's AI assistance shines. With an API key provided, it generates contextually appropriate evaluation tasks and graders based on your skill's purpose. Without a key, you receive a well-commented template to customize manually. Force overwrite existing configurations with --force.

Pro tip: The API key used here also determines your default agent. GEMINI_API_KEY routes to Gemini, ANTHROPIC_API_KEY to Claude, OPENAI_API_KEY to Codex. This auto-detection eliminates configuration friction.

Configuration Editing

Open the generated eval.yaml and customize for your specific skill. The structure supports global defaults, per-task overrides, workspace file staging, and grader composition. We'll examine the full schema in the code examples below.

Running Your First Evaluation

Execute with the smoke test preset for rapid feedback:

GEMINI_API_KEY=your-key skillgrade --smoke

This runs 5 trials—enough to catch obvious failures without consuming excessive API quota. Progress to --reliable (15 trials) for statistical confidence, or --regression (30 trials) before major releases.

Reviewing Results

Two preview modes serve different needs:

skillgrade preview          # Terminal-optimized report
skillgrade preview browser  # Interactive web UI at http://localhost:3847

Results persist to $TMPDIR/skillgrade/<skill-name>/results/ by default. Override with --output=DIR for artifact collection in CI pipelines.

CI Integration

For GitHub Actions or similar, use local provider execution to eliminate Docker overhead in already-ephemeral runners:

# .github/workflows/skillgrade.yml
- run: |
    npm i -g skillgrade
    cd skills/superlint
    GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }} skillgrade --regression --ci --provider=local

The --ci flag ensures non-zero exit if pass rate falls below --threshold (default 0.8), failing the build exactly when quality degrades.

REAL Code Examples from the Repository

Let's dissect three authentic patterns from Skillgrade's documentation, with detailed commentary on production implementation.

Example 1: Complete eval.yaml Configuration

This comprehensive example demonstrates the full configuration power—from defaults to per-task overrides, workspace staging to grader composition:

version: "1"

# Optional: explicit path to skill directory (defaults to auto-detecting SKILL.md)
# skill: path/to/my-skill

defaults:
  agent: gemini          # gemini | claude | codex | acp
  provider: docker       # docker | local
  trials: 5
  timeout: 300           # seconds before task abortion
  threshold: 0.8         # minimum pass rate for --ci mode success
  grader_model: gemini-3-flash-preview  # default for LLM rubric evaluation
  acp:                   # ACP agent configuration (optional)
    command: gemini --acp  # command to start ACP-compatible agent
    env:                  # optional environment variables for agent process
      DEBUG: "1"
  docker:
    base: node:20-slim   # container base image; customize for your runtime needs
    setup: |             # extra commands run during image build
      apt-get update && apt-get install -y jq
  environment:           # container resource limits for reproducible performance
    cpus: 2
    memory_mb: 2048

tasks:
  - name: fix-linting-errors
    instruction: |
      Use the superlint tool to fix coding standard violations in app.js.

    workspace:                           # files copied into container before execution
      - src: fixtures/broken-app.js      # source in your repo
        dest: app.js                     # destination path in container
      - src: bin/superlint               # binary to install
        dest: /usr/local/bin/superlint   # system path for command availability
        chmod: "+x"                      # ensure executable permissions

    graders:
      - type: deterministic
        setup: npm install typescript    # grader-specific dependencies installed pre-evaluation
        run: npx ts-node graders/check.ts  # command producing JSON score output
        weight: 0.7                      # 70% of final score: did it work correctly?
      - type: llm_rubric
        rubric: |
          Did the agent follow the check → fix → verify workflow?
        model: gemini-2.0-flash          # optional model override for this grader
        weight: 0.3                      # 30% of final score: was the approach sound?

    # Per-task overrides (optional) — demonstrate when specific tasks need different handling
    agent: claude                        # use Claude specifically for this complex task
    trials: 10                           # more trials for higher confidence on critical path
    timeout: 600                         # extended timeout for complex linting scenarios

Key insight: The workspace stanza is deceptively powerful. By staging fixtures/broken-app.js as app.js, you create realistic test conditions without modifying your repository structure. The chmod: "+x" pattern ensures binaries are executable—easy to forget, critical for functionality. Per-task overrides let you tune agent selection and resource allocation precisely, rather than forcing global compromises.

Example 2: Deterministic Grader Bash Implementation

For objective evaluations, deterministic graders run commands and parse structured JSON. Here's the README's complete bash example, annotated for production use:

#!/bin/bash
# deterministic-grader.sh — Production-ready deterministic grader for file operations

# Initialize counters for score calculation
passed=0; total=2
c1_pass=false c1_msg="File missing"
c2_pass=false c2_msg="Content wrong"

# Check 1: Verify output file existence
if test -f output.txt; then
  passed=$((passed + 1))
  c1_pass=true
  c1_msg="File exists"
fi

# Check 2: Verify file contains expected content
# Redirect stderr to /dev/null to prevent grep errors from polluting output
if grep -q "expected" output.txt 2>/dev/null; then
  passed=$((passed + 1))
  c2_pass=true
  c2_msg="Content correct"
fi

# Calculate score with awk — bc unavailable in node:20-slim base image
# Format to 2 decimal places for consistent JSON schema compliance
score=$(awk "BEGIN {printf \"%.2f\", $passed/$total}")

# Output strictly conforming JSON schema that Skillgrade parses
# 'score' (0.0-1.0) and 'details' are required; 'checks' array optional but valuable for debugging
echo "{\"score\":$score,\"details\":\"$passed/$total passed\",\"checks\":[{\"name\":\"file\",\"passed\":$c1_pass,\"message\":\"$c1_msg\"},{\"name\":\"content\",\"passed\":$c2_pass,\"message\":\"$c2_msg\"}]}"

Critical implementation note: The README explicitly warns to use awk for arithmetic since bc isn't available in the default node:20-slim image. This detail exemplifies Skillgrade's production maturity—the authors have already hit and documented the sharp edges so you don't have to. The structured checks array, while optional, transforms debugging from guesswork into precise failure diagnosis.

Example 3: LLM Rubric Grader for Qualitative Evaluation

When objective checks aren't enough, LLM rubric graders evaluate behavioral and qualitative dimensions:

graders:
  - type: llm_rubric
    rubric: |
      Workflow Compliance (0-0.5):
      - Did the agent follow the mandatory 3-step workflow?
      - Were all prerequisite checks executed before modifications?

      Efficiency (0-0.5):
      - Completed in ≤5 commands?
      - No redundant or exploratory operations?
    weight: 0.3
    model: gemini-2.0-flash    # optional; auto-detected from API key if omitted

Why this matters: Many agent failures aren't functional—they're procedural. An agent that eventually produces correct output but took twenty destructive commands to get there is not a success in production. LLM rubric graders catch these "success failures" by evaluating the session transcript against structured criteria. The explicit point allocation (0-0.5 per category) creates consistent, defensible scoring that human reviewers can audit and adjust.

Advanced Usage & Best Practices

Validate Before You Evaluate. The --validate flag runs graders against reference solutions, confirming your evaluation logic itself is sound. A broken grader is worse than no grader—it provides false confidence. Always validate new grader configurations before incorporating into CI.

Grade Outcomes, Not Steps. The README's first best practice is deceptively simple: check that the file was fixed, not that the agent ran eslint --fix. Implementation details change; requirements evolve. Outcome-based grading survives refactoring. Step-based grading breaks constantly.

Explicit Output Naming. If your grader checks for output.html, your instruction must specify output.html. Agents cannot infer filenames from grader logic—they only see instructions. This coordination between instruction and evaluation is where many first-time users stumble.

Progressive Evaluation Depth. Start with --smoke during active development for rapid iteration. Escalate to --reliable before peer review, and --regression before any production deployment. This tiered approach balances speed against confidence appropriately.

Resource Consciousness. The environment configuration lets you cap CPU and memory. Use this aggressively—agents can consume surprising resources, and reproducible evaluation requires reproducible constraints. The default 2 CPUs and 2048 MB is a starting point, not a universal prescription.

File Reference Convenience. String values for instruction, rubric, and run automatically resolve file paths. Store complex rubrics in rubrics/quality.md, lengthy instructions in instructions/fix-linting.md. This keeps eval.yaml readable while supporting sophisticated evaluation logic.

Comparison with Alternatives

Skillgrade's integrated approach eliminates the glue code and context switching that fragment alternative approaches. Where manual testing is non-repeatable, custom scripts are unmaintainable, and academic benchmarks are impractical for production workflows, Skillgrade provides a purpose-built, cohesive solution.

FAQ

What exactly is a "skill" in this context? A skill is a structured capability definition—typically a SKILL.md file—that teaches AI agents how to perform specific tasks in your codebase. It includes tool descriptions, usage patterns, and contextual guidance that helps agents understand when and how to apply the capability.

Do I need Docker if I'm only running local evaluations? No. Use --provider=local to execute directly on your machine. Docker is recommended for development to protect against potentially destructive agent actions, but CI environments and trusted local testing can safely use local execution.

Which AI agents does Skillgrade support? Currently Gemini, Claude (Anthropic), Codex (OpenAI), OpenCode with multiple subagents, and any ACP-compatible agent. The framework is architected for extensibility—new agents require implementing a standardized interface.

How much do evaluations cost in API usage? Costs scale with trial count, grader complexity, and model selection. A --smoke run with 5 trials uses minimal tokens. --regression with 30 trials and LLM rubric graders is proportionally more expensive. The deterministic grader type eliminates LLM costs for objective checks.

Can I use different models for agent execution and grader evaluation? Absolutely. The agent field controls which model executes tasks, while grader_model or per-grader model overrides control evaluation. This lets you test if cheap models can execute skills that expensive models grade, optimizing cost-performance tradeoffs.

What happens if an agent hangs or runs indefinitely? The timeout configuration (default 300 seconds) aborts stuck tasks. Failed tasks contribute zero to pass rate calculations, ensuring hung executions don't silently corrupt your metrics.

Is my API key secure in evaluation logs? Yes. All environment variable values are redacted from persisted session logs. Keys loaded from .env files receive identical protection. Shell-provided values override .env for flexibility without compromising security.

Conclusion

The AI agent ecosystem is maturing rapidly, but maturity without measurement is just chaos with better marketing. Skillgrade represents a critical evolution—applying the hard-won lessons of software testing to the novel challenges of agent evaluation. It's not the only tool you'll need, but it might be the most important one you've been missing.

I've seen too many agent skills fail in production because nobody systematically asked "does this actually work?" Skillgrade makes that question impossible to ignore and effortless to answer. The initialization assistance gets you started, the dual grading architecture covers both objective and qualitative dimensions, and the CI integration ensures quality gates persist.

My recommendation? Install Skillgrade today. Run skillgrade init on your most important skill. Watch the first evaluation results—wince at the failures, celebrate the passes, and iterate. Within a week, you'll have confidence levels you didn't think possible in agent development. Within a month, you'll wonder how anyone ships skills without this.

The repository is waiting. Your skills deserve better than hope. Get Skillgrade on GitHub and start testing like your users' experience depends on it—because it does.

Ready to level up your agent development? Star mgechev/skillgrade, try it on your first skill, and join the growing community of developers who refuse to ship broken agent capabilities.