Stop Wrestling the Official ClawHub API This Layer Exposes Everything

Stop Wrestling the Official ClawHub API—This Layer Exposes Everything

What if I told you that 36,000+ developer skills are sitting behind an API that deliberately hides the data you actually need? No full-text search. No security scan results. No way to peek inside a skill's source code without downloading the entire package. If you've been building on top of ClawHub—the skills marketplace that's quietly becoming the App Store for AI agents—you've felt this pain. You've written hacky workarounds. You've considered querying their Convex database directly (don't). You've probably stared at rate limit errors wondering why the "official" way feels so… incomplete.

Here's the secret top builders already know: there's a shadow layer that turns ClawHub's locked-down data into a fully queryable, searchable, security-scanned REST API. It's called ClawHub Layer API, and it's about to change how you discover, evaluate, and integrate skills into your AI workflows. Built by the team at AtomicBot, this open-source project doesn't just wrap the official API—it replaces the entire data pipeline, pulling complete catalog data from ClawHub's Convex backend, caching it in MongoDB, and serving it through clean, documented REST endpoints with features the official API will never expose.

In this deep dive, I'll show you exactly what you're missing, how to deploy your own instance in under 5 minutes, and why developers are already abandoning direct ClawHub integration for this superior alternative. The repository is live at github.com/AtomicBot-ai/clawhub-layer-api—but trust me, you'll want to understand why it exists before you clone a single byte.

What Is ClawHub Layer API?

ClawHub Layer API is an open-source REST API layer built in NestJS that transforms ClawHub's restricted skill marketplace data into a fully accessible, queryable, and cacheable API surface. Created by AtomicBot-ai, it addresses a critical gap: the official ClawHub API is intentionally limited, forcing developers who need complete data to either scrape (fragile, against ToS) or query the Convex database directly (complex, requires internal knowledge).

The project has gained rapid traction because it solves a genuine infrastructure problem in the AI agent ecosystem. ClawHub functions as a marketplace where developers publish "skills"—reusable capabilities for AI agents, similar to how npm packages work for JavaScript^{↗ Bright Coding Blog} or PyPI for Python^{↗ Bright Coding Blog}. But unlike mature package ecosystems, ClawHub's official API exposes only surface-level metadata. Want to know if a skill contains malicious code? The official API won't tell you. Need to search across skill descriptions with weighted relevance? Not supported. Want to read a skill's SKILL.md documentation without installing it? Impossible.

ClawHub Layer API changes this equation entirely. It operates as a synchronization and caching layer: a scheduled job (configurable via cron) pages through ClawHub's complete catalog using internal Convex queries, persists everything to MongoDB, and enriches records on-demand with detail data, security scans, file contents, and live comments. The result is a self-hosted API that serves data faster, more completely, and more flexibly than the official endpoints ever could.

Built on modern infrastructure—NestJS for the API framework, MongoDB for document caching, Docker^{↗ Bright Coding Blog} for deployment, and Swagger for interactive documentation—it's designed for production use by teams building serious agent platforms. The live public instance at clawhub.atomicbot.ai demonstrates its capabilities, but the real power comes from running your own instance with custom sync schedules, cache policies, and data retention rules.

Key Features That Expose Hidden Data

What makes developers switch? Let's dissect the capabilities that simply don't exist elsewhere:

📦 Complete Catalog Access (36,000+ Skills) — The bulk sync engine pages through listPublicPageV4, capturing every skill with full metadata: download counts, star ratings, install statistics, version histories, tags, and ownership chains. This isn't paginated API calls that timeout—it's a complete mirror updated every 3 hours by default.

🔍 Weighted Full-Text Search — The search endpoint implements intelligent relevance scoring: skill slugs match at 10× weight, names at 5×, and descriptions at 1×. This means searching for "summarize" surfaces summarize-pdf before a skill that merely mentions summarization in its description. The official API has no search capability whatsoever.

🛡️ Security Intelligence Layer — Here's where it gets serious. Every skill gets VirusTotal scan results and LLM-based security analysis. In an ecosystem where you're executing third-party code in your AI agents, knowing whether a skill has been flagged for malicious behavior isn't optional—it's existential. The official API exposes zero security data.

🚩 Moderation Transparency — Skills get flagged. Skills get removed. The official API pretends this doesn't happen. ClawHub Layer API exposes suspicious/malicious flags, removal status, and reason codes, so you can build trust scoring into your skill selection logic.

📄 Arbitrary File Access — Need to inspect a skill's package.json before installing? Want to read SKILL.md documentation programmatically? The file content endpoint returns raw files from any skill package without downloading the entire artifact. This enables dynamic documentation rendering, dependency analysis, and automated code review pipelines.

💬 Live Comment Threads — Community sentiment matters. The comments endpoint pulls live discussions with full user profiles, enabling reputation systems and social proof integration.

🔀 Fork Relationship Mapping — Skills fork. Skills merge. Understanding provenance helps identify canonical implementations versus abandoned copies. The fork tracking endpoint reveals these relationships.

⚡ Smart Caching Architecture — Detail data and file contents are fetched on first request and cached with configurable TTL (default: 3 hours). This balances freshness with performance—hot skills stay cached, cold skills get enriched when needed.

Real-World Use Cases Where This Shines

AI Agent Platforms Building Skill Marketplaces

You're building the next big AI agent framework. Your users want to discover and install skills without leaving your platform. The official API gives you names and descriptions—barely enough for a directory listing. ClawHub Layer API gives you searchable, filterable, security-scored skill catalogs with inline documentation rendering. Your users make informed decisions; you reduce support burden from malicious skill installations.

Security-Focused CI/CD Pipelines

Every skill your agents execute is arbitrary code. Before deployment, your pipeline needs to verify: Has VirusTotal flagged this? What's the LLM security assessment? Has moderation actioned it? The security and moderation endpoints enable automated gatekeeping—block skills with suspicious flags, quarantine unanalyzed skills, audit trail everything.

Competitive Intelligence & Research

Analyzing the skills ecosystem for investment, acquisition, or research? You need complete data: download trends, fork relationships, comment sentiment, version evolution. The official API's fragmentary data makes meaningful analysis impossible. The Layer API's complete mirror enables time-series analysis, network graph construction, and market gap identification.

Documentation & Developer Experience Tools

Building an IDE plugin or CLI tool for skill management? Users expect to preview documentation, inspect dependencies, and understand security posture before installation. The file content and security endpoints let you render rich skill cards with live documentation, dependency trees, and trust badges—experiences impossible with official APIs.

Step-by-Step Installation & Setup Guide

Ready to escape API limitations? Here's your complete deployment path.

Prerequisites

• Docker 20.10+ and Docker Compose v2+
• 4GB RAM minimum (MongoDB + NestJS + caching overhead)
• Optional: MongoDB Atlas URI if avoiding local database

Production Deployment (Recommended)

# Clone the repository
git clone https://github.com/AtomicBot-ai/clawhub-layer-api.git
cd clawhub-layer-api

# Launch with production configuration
docker compose -f docker-compose.prod.yml up -d

This starts:

• MongoDB container with persistent volume
• NestJS app container on port 3000
• Network bridge for inter-service communication

Verify health:

curl http://localhost:3000/health
# Expected: {"status":"ok"} or similar health payload

Access interactive documentation at http://localhost:3000/docs—the Swagger UI with complete endpoint specifications and test interface.

Trigger Initial Data Sync

The container starts with empty caches. Populate it:

# Execute sync command inside running container
docker compose -f docker-compose.prod.yml exec app node dist/cli sync

Critical timing note: The initial sync processes ~36,000 skills in 2-3 minutes. Monitor with:

docker compose -f docker-compose.prod.yml logs -f app

You'll see pagination progress through listPublicPageV4 and MongoDB upsert confirmations.

Development Environment (Hot-Reload)

For contribution or customization:

docker compose up -d

This mounts source code with nodemon hot-reload, rebuilds on file changes, and uses development-optimized settings. MongoDB starts automatically; no external database required.

Configuration Customization

Create .env from the documented variables:

Pro tip: For high-availability deployments, run multiple app containers behind a load balancer with shared MongoDB. The stateless design makes horizontal scaling trivial.

REAL Code Examples from the Repository

Let's examine actual patterns from the codebase, adapted for clarity and educational value.

Example 1: Production Docker Compose Deployment

The docker-compose.prod.yml orchestrates the complete stack. Here's the essence:

# docker-compose.prod.yml (conceptual structure)
version: '3.8'

services:
  mongodb:
    image: mongo:6-jammy
    restart: unless-stopped
    volumes:
      - mongodb_data:/data/db  # Persistent storage survives container restarts
    environment:
      MONGO_INITDB_DATABASE: clawhub-layer  # Pre-creates target database

  app:
    build:
      context: .
      dockerfile: Dockerfile
    restart: unless-stopped
    ports:
      - "${PORT:-3000}:3000"  # Configurable via environment
    environment:
      MONGODB_URI: mongodb://mongodb:27017/clawhub-layer  # Service DNS resolution
      # Additional env vars loaded from .env or compose environment
    depends_on:
      - mongodb  # Ensures database readiness before app start

volumes:
  mongodb_data:  # Named volume for data persistence

Key architectural decisions revealed: MongoDB runs as a co-located service rather than external dependency, simplifying single-node deployment. The depends_on ensures startup ordering. Port binding uses shell parameter expansion for flexibility. Volume naming prevents accidental data loss on docker compose down.

Example 2: CLI Sync Command Implementation

The sync functionality demonstrates the data pipeline pattern:

# Direct execution inside container
node dist/cli sync

# Or from host via docker compose exec
docker compose -f docker-compose.prod.yml exec app node dist/cli sync

This triggers the bulk synchronization engine that:

1. Authenticates with ClawHub's Convex deployment using internal query endpoints
2. Paginates through listPublicPageV4 with cursor-based iteration
3. Transforms raw Convex documents into normalized MongoDB schemas
4. Upserts records with atomic operations (insert new, update existing, preserve cached detail data)
5. Reports progress via structured logging

The exec pattern is crucial for container-native operations—no need to install Node.js locally, no version conflicts, complete environment consistency.

Example 3: API Endpoint Consumption Patterns

Here are practical client implementations against the Layer API:

# --- List skills with security filtering ---
# Returns paginated catalog, sorted by downloads, excluding flagged content
curl "https://clawhub.atomicbot.ai/api/skills?page=1&limit=25&sort=downloads&dir=desc&nonSuspiciousOnly=true" \
  -H "Accept: application/json" | jq '.skills[] | {slug, downloads, securityScore}'

# --- Weighted full-text search ---
# "summarize" matches slug (10x), name (5x), description (1x)
curl "https://clawhub.atomicbot.ai/api/skills/search?q=summarize&limit=25" \
  -H "Accept: application/json" | jq '.results[] | {slug, name, relevanceScore}'

# --- Complete skill intelligence ---
# Single call returns: metadata + versions + security analysis + moderation + files + comments
curl "https://clawhub.atomicbot.ai/api/skills/pdf-summarizer-pro" \
  -H "Accept: application/json" | jq '{
    slug: .slug,
    owner: .owner.githubUsername,
    latestVersion: .versions[0].version,
    virusTotal: .security.virusTotal.lastAnalysisStats,
    llmAnalysis: .security.llmAssessment.riskLevel,
    isRemoved: .moderation.removed,
    fileCount: .files | length,
    commentCount: .comments.total
  }'

# --- Read skill documentation without installation ---
# Returns raw SKILL.md content for rendering or analysis
curl "https://clawhub.atomicbot.ai/api/skills/pdf-summarizer-pro/files?path=SKILL.md" \
  -H "Accept: text/markdown↗ Smart Converter"  # or application/json for structured response

# --- Community sentiment analysis ---
# Live comments with user profiles for reputation scoring
curl "https://clawhub.atomicbot.ai/api/skills/pdf-summarizer-pro/comments?limit=50" \
  -H "Accept: application/json" | jq '.comments[] | {author: .user.githubUsername, body: .text, createdAt}'

The power of composition: These endpoints combine to enable workflows impossible with official APIs. Example: a pre-installation safety report that checks VirusTotal results, moderation status, and community sentiment before executing a single line of third-party code.

Example 4: Programmatic Integration (JavaScript/TypeScript)

// Modern fetch-based client with TypeScript interfaces
interface SkillDetail {
  slug: string;
  name: string;
  security: {
    virusTotal: {
      lastAnalysisStats: {
        malicious: number;
        suspicious: number;
        undetected: number;
      };
    };
    llmAssessment: {
      riskLevel: 'low' | 'medium' | 'high' | 'critical';
      summary: string;
    };
  };
  moderation: {
    removed: boolean;
    reasonCode?: string;
  };
}

class ClawHubLayerClient {
  private baseUrl: string;

  constructor(baseUrl: string = 'https://clawhub.atomicbot.ai') {
    this.baseUrl = baseUrl.replace(/\/$/, ''); // Normalize trailing slash
  }

  async getSkillWithSecurityCheck(slug: string): Promise<SkillDetail | null> {
    const response = await fetch(`${this.baseUrl}/api/skills/${encodeURIComponent(slug)}`);
    
    if (response.status === 404) return null;
    if (!response.ok) throw new Error(`API error: ${response.status}`);
    
    const skill: SkillDetail = await response.json();
    
    // Client-side safety gate
    if (skill.moderation?.removed) {
      console.warn(`SKIPPED: ${slug} is removed (${skill.moderation.reasonCode})`);
      return null;
    }
    if (skill.security?.virusTotal?.lastAnalysisStats?.malicious > 0) {
      console.warn(`QUARANTINE: ${slug} has VirusTotal detections`);
      // Could throw, return with warning flag, or auto-reject
    }
    
    return skill;
  }

  async *searchSkills(query: string, batchSize: number = 25): AsyncGenerator<SkillDetail> {
    let page = 1;
    while (true) {
      const response = await fetch(
        `${this.baseUrl}/api/skills/search?q=${encodeURIComponent(query)}&limit=${batchSize}&page=${page}`
      );
      const data = await response.json();
      
      if (data.results.length === 0) break;
      
      for (const result of data.results) {
        yield result;
      }
      
      page++;
      if (data.results.length < batchSize) break; // Last page
    }
  }
}

// Usage: stream-process all "summarize" skills with safety checks
const client = new ClawHubLayerClient();
for await (const skill of client.searchSkills('summarize')) {
  const safeSkill = await client.getSkillWithSecurityCheck(skill.slug);
  if (safeSkill) {
    // Proceed with integration
  }
}

This demonstrates defensive programming patterns: URL encoding, graceful 404 handling, security gate integration, and memory-efficient async iteration for large result sets.

Advanced Usage & Best Practices

Cache Warming Strategy: Don't wait for user requests to trigger detail enrichment. After bulk sync, hit critical skills proactively:

# Warm cache for top-downloaded skills
for slug in $(curl -s "https://clawhub.atomicbot.ai/api/skills?sort=downloads&limit=100" | jq -r '.skills[].slug'); do
  curl -s "https://clawhub.atomicbot.ai/api/skills/${slug}" > /dev/null &
done
wait

Custom Sync Scheduling: For event-driven architectures, disable cron and trigger sync via webhook:

# In your CI/CD or event handler
curl -X POST http://your-instance:3000/admin/trigger-sync \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

MongoDB Index Optimization: The default indexes handle most queries, but add compound indexes for your access patterns:

// For frequent owner + sort queries
db.skills.createIndex({ "owner.githubUsername": 1, "stats.downloads": -1 })

Monitoring Integration: The /health endpoint provides basic liveness. Extend with custom metrics:

# Prometheus-style metrics (implement in fork)
curl http://localhost:3000/metrics
# clawhub_sync_last_success_timestamp 1704067200
# clawhub_cache_hit_ratio 0.847

Comparison with Alternatives

The verdict: Direct Convex queries offer raw power but require deep platform knowledge, internal endpoint documentation, and ongoing maintenance as schemas evolve. The official API is stable but crippled. ClawHub Layer API hits the sweet spot—complete data, reasonable setup, production-grade architecture, and community-maintained compatibility.

FAQ: What Developers Ask

Is this officially supported by ClawHub? No—it's an independent open-source project. However, it uses public Convex query patterns and respects rate limits. The maintainers actively monitor for breaking changes.

Can I get banned for using this? The public instance at clawhub.atomicbot.ai handles all ClawHub interaction. Your self-hosted instance makes requests to your own MongoDB cache. The sync process is polite (configurable delays, reasonable cron intervals).

How fresh is the data? Default sync is every 3 hours; detail cache TTL is 3 hours. For most use cases, 3-6 hour staleness is acceptable. Reduce SYNC_CRON and CACHE_TTL_HOURS for near-real-time at higher resource cost.

Does it handle private skills? No—only publicly listed skills in ClawHub's catalog. Private or unlisted skills require authentication ClawHub doesn't expose to third parties.

What's the performance with 36,000+ skills? MongoDB handles this comfortably with proper indexing. Typical queries: list (<50ms), search (<200ms), detail from cache (<20ms), detail with enrichment (<2s first time).

Can I contribute or fork? Absolutely. ISC license permits commercial use, modification, and distribution. The codebase is standard NestJS—familiar to most Node.js developers.

How do I report security issues in skills? The API exposes existing ClawHub security data but doesn't accept new reports. Use ClawHub's official channels for skill security disclosures.

Conclusion: Your Data, Your Rules

The AI agent ecosystem is exploding, but infrastructure maturity lags far behind the hype. ClawHub Layer API represents a critical evolution—transforming a black-box marketplace into transparent, queryable, secure infrastructure that serious builders can depend on.

I've shown you the gaps in official APIs, the architecture that fills them, and the exact commands to deploy your own instance. The repository at github.com/AtomicBot-ai/clawhub-layer-api is actively maintained, well-documented, and ready for production workloads. Whether you're building the next agent framework, hardening your CI/CD pipeline, or simply tired of incomplete data, this is your exit ramp from API frustration.

Stop accepting partial data. Stop building on opaque infrastructure. Deploy ClawHub Layer API today and finally see everything.

Star the repo, open an issue with your use case, or fork it for your custom needs. The team at AtomicBot is building in public—join them.