Stop Writing Regex Scrapers! Use llm-scraper Instead

Stop Writing Regex Scrapers! Use llm-scraper Instead

What if I told you that your beautiful, meticulously crafted web scraper is already broken?

You spent hours perfecting those CSS selectors. You battled through CAPTCHAs, dynamic rendering, and sites that change their DOM structure on a whim. You celebrated when it finally extracted that product price—only to wake up to a Slack alert at 3 AM because the website redesigned overnight and your "bulletproof" scraper is now returning null across 47,000 records.

Here's the brutal truth: traditional web scraping is a losing battle. Every selector you write is technical debt. Every regex pattern is a ticking time bomb. The modern web is dynamic, JavaScript^{↗ Bright Coding Blog}-heavy, and increasingly hostile to conventional extraction methods.

But what if you could simply describe what you want—in plain English, with type-safe schemas—and let an intelligent system figure out how to get it?

Enter llm-scraper, the open-source TypeScript library that's making traditional scrapers obsolete. Created by developer Mish Ushakov, this tool doesn't just scrape web pages—it understands them. By combining the power of Large Language Models with the reliability of Playwright and Zod schemas, llm-scraper transforms the chaotic, unstructured web into clean, typed data structures.

And the best part? It's not locked to one AI provider. GPT-4o, Claude 3.5 Sonnet, Gemini, Llama, Qwen—you choose your weapon.

Ready to never write another brittle XPath query again? Let's dive deep.

What is llm-scraper?

llm-scraper is a TypeScript library that extracts structured data from any webpage using Large Language Models. At its core, it's a radical reimagining of what web scraping can be in the age of AI.

Unlike traditional scraping frameworks that rely on fragile DOM traversal, llm-scraper leverages LLMs to comprehend page content and extract exactly what you need—defined through strongly-typed schemas. It sits at the intersection of browser automation (via Playwright), schema validation (via Zod), and modern AI SDKs (via Vercel AI SDK 6).

The creator, Mish Ushakov, built this tool with a clear vision: make web data extraction as simple as defining a TypeScript interface. The project has gained significant traction in the developer community, particularly after its 2.0 release which brought Vercel AI SDK 6 support and modernized examples.

Why it's trending now:

• The death of reliable selectors: Modern SPAs, shadow DOM, and anti-scraping measures have made traditional methods prohibitively expensive to maintain
• LLM cost collapse: GPT-4o and Gemini Flash are now cheap enough for high-volume extraction tasks
• TypeScript's dominance: Developers demand type safety, and llm-scraper delivers it natively
• Multi-modal capabilities: Screenshot-based extraction opens entirely new possibilities for visual content

The library's philosophy is deceptively simple: you describe the shape of your data, the AI finds it. No more inspecting element hierarchies. No more praying that div.product-card:nth-child(3) > span.price survives the next deployment.

Key Features That Change Everything

Multi-Provider LLM Support

llm-scraper doesn't lock you into a single AI ecosystem. It supports the entire landscape of modern LLMs through the Vercel AI SDK:

• OpenAI: GPT-4o, GPT-4o-mini for cost-efficient extraction
• Anthropic: Claude 3.5 Sonnet for superior reasoning on complex layouts
• Google: Gemini 1.5 Flash with massive context windows
• Groq: Llama models with blazing inference speeds
• Ollama: Self-hosted Llama for privacy-sensitive deployments

This flexibility means you can optimize for cost, speed, or accuracy depending on your use case.

Schema-First Architecture with Zod

Forget parsing HTML strings. Define your data structure with Zod schemas or JSON Schema, and get full TypeScript inference:

const schema = z.object({
  products: z.array(z.object({
    name: z.string(),
    price: z.number(),
    inStock: z.boolean()
  }))
})
// TypeScript knows exactly what `data` contains
const { data } = await scraper.run(page, Output.object({ schema }))

Six Formatting Modes for Maximum Flexibility

The library adapts to your content, not the other way around:

Streaming Objects for Real-Time Applications

Don't wait for the full extraction. Stream partial results as the LLM generates them—critical for responsive UIs and large datasets.

Code Generation: The Secret Weapon

Here's where it gets insane. llm-scraper can generate reusable Playwright scripts from your schemas. Extract once with AI, then run lightning-fast deterministic scrapers forever:

const { code } = await scraper.generate(page, Output.object({ schema }))
// `code` is a Playwright script you can execute without LLM calls
const result = await page.evaluate(code)

This hybrid approach combines AI flexibility with traditional performance—best of both worlds.

Real-World Use Cases Where llm-scraper Dominates

1. Competitive Intelligence at Scale

Your pricing team needs real-time competitor data. Traditional scrapers break when sites A/B test layouts. With llm-scraper, you define:

const pricingSchema = z.object({
  competitor: z.string(),
  products: z.array(z.object({
    name: z.string(),
    currentPrice: z.number(),
    originalPrice: z.number().optional(),
    discount: z.string().optional(),
    availability: z.enum(['in_stock', 'out_of_stock', 'pre_order'])
  }))
})

The LLM handles variant layouts, dynamic loading, and even interprets "Was $99.99, Now $79.99" into structured numbers.

2. Financial Document Processing

SEC filings, earnings reports, press releases—unstructured financial text that analysts spend hours parsing. llm-scraper with format: 'text' and Readability.js preprocessing extracts key metrics:

• Revenue figures with fiscal periods
• Risk factor counts and sentiment
• Executive compensation tables
• Forward-looking statements

3. E-Commerce Catalog Migration

Migrating 50,000 products from a legacy platform? The source site has inconsistent HTML, missing fields, and multilingual content. Define your target schema once, let the LLM handle the mess:

const productSchema = z.object({
  title: z.string(),
  description: z.string().optional(),
  specifications: z.record(z.string()), // Flexible key-value pairs
  images: z.array(z.string().url()),
  variants: z.array(z.object({
    sku: z.string(),
    attributes: z.record(z.string())
  })).optional()
})

4. Research & Academic Data Collection

Academic papers, preprints, and conference proceedings live across hundreds of formats. llm-scraper's markdown mode normalizes everything:

• Extract citation networks from bibliography sections
• Identify methodology sections across different heading structures
• Correlate figures and tables with their in-text references

5. Social Media^{↗ Bright Coding Blog} Monitoring (Ethical)

Public posts, forum discussions, review sites—highly dynamic content with nested replies and varying structures. The image mode enables multi-modal analysis of screenshot-based feeds where DOM access is restricted.

Step-by-Step Installation & Setup Guide

Prerequisites

• Node.js 18+ (for native fetch and modern ESM)
• A valid API key for your chosen LLM provider

Step 1: Install Core Dependencies

npm i zod playwright llm-scraper

Why these three?

• zod: Runtime schema validation with TypeScript inference
• playwright: Browser automation for JavaScript-rendered content
• llm-scraper: The orchestration layer

Step 2: Install Your LLM Provider

For OpenAI (most popular):

npm i @ai-sdk/openai

For Anthropic Claude:

npm i @ai-sdk/anthropic

For Google Gemini:

npm i @ai-sdk/google

For Groq (fast Llama inference):

npm i @ai-sdk/openai  # Uses OpenAI-compatible API

For local Ollama:

npm i ollama-ai-provider-v2

Step 3: Configure Environment Variables

Create a .env file:

# Required for your chosen provider
OPENAI_API_KEY=sk-your-key-here
# ANTHROPIC_API_KEY=sk-ant-your-key
# GOOGLE_GENERATIVE_AI_API_KEY=your-key
# GROQ_API_KEY=gsk-your-key

Step 4: Initialize Your LLM Instance

OpenAI:

import { openai } from '@ai-sdk/openai'
const llm = openai('gpt-4o')  // or 'gpt-4o-mini' for cost savings

Anthropic:

import { anthropic } from '@ai-sdk/anthropic'
const llm = anthropic('claude-3-5-sonnet-20240620')

Google:

import { google } from '@ai-sdk/google'
const llm = google('gemini-1.5-flash')

Groq (with custom baseURL):

import { createOpenAI } from '@ai-sdk/openai'
const groq = createOpenAI({
  baseURL: 'https://api.groq.com/openai/v1',
  apiKey: process.env.GROQ_API_KEY,
})
const llm = groq('llama3-8b-8192')

Ollama (self-hosted):

import { ollama } from 'ollama-ai-provider-v2'
const llm = ollama('llama3')  // Runs locally, zero API costs

Step 5: Create the Scraper Instance

import LLMScraper from 'llm-scraper'

const scraper = new LLMScraper(llm)

That's it. You're ready to extract structured data from any webpage.

REAL Code Examples from the Repository

Let's examine the actual implementation patterns from llm-scraper's documentation, with detailed explanations of what makes each powerful.

Example 1: HackerNews Extraction (The Classic Demo)

This is the repository's flagship example—extracting top stories from Hacker News. Study it carefully; it demonstrates the complete lifecycle:

import { chromium } from 'playwright'
import { z } from 'zod'
import { Output } from 'ai'
import { openai } from '@ai-sdk/openai'
import LLMScraper from 'llm-scraper'

// Launch a browser instance — Playwright handles JS-rendered content
const browser = await chromium.launch()

// Initialize LLM provider — GPT-4o for reliable structured extraction
const llm = openai('gpt-4o')

// Create a new LLMScraper with our configured LLM
const scraper = new LLMScraper(llm)

// Open new page and navigate to target
const page = await browser.newPage()
await page.goto('https://news.ycombinator.com')

// Define Zod schema: exactly what we want extracted
const schema = z.object({
  top: z
    .array(
      z.object({
        title: z.string(),      // Story headline
        points: z.number(),     // Upvote count as number
        by: z.string(),         // Username who submitted
        commentsURL: z.string(), // Link to discussion
      })
    )
    .length(5)  // Force exactly 5 results — LLM will paginate/filter
    .describe('Top 5 stories on Hacker News'), // Context for the LLM
})

// Execute extraction: HTML mode for clean, processed markup
const { data } = await scraper.run(page, Output.object({ schema }), {
  format: 'html',
})

// data.top is fully typed: { title: string, points: number, ... }[]
console.log(data.top)

// Cleanup resources — always close browser to prevent memory leaks
await page.close()
await browser.close()

What makes this powerful:

Notice how we never inspect the HN DOM. We don't care that stories are in <tr class="athing"> or that scores live in <span id="score_...">. The LLM understands the visual structure and maps it to our schema. If HN redesigns tomorrow, this code keeps working.

The .describe() on the schema array is critical—it gives the LLM semantic context. Without it, you might get 5 random stories. With it, the LLM understands "top" means highest-ranked.

Example 2: Streaming for Real-Time Applications

When you need results now, not after the full generation completes:

// Same setup as before: browser, llm, scraper, page, schema...

// Use .stream() instead of .run() for incremental results
const { stream } = await scraper.stream(page, Output.object({ schema }))

// Iterate through partial objects as LLM generates them
for await (const partialData of stream) {
  // partialData.top might be incomplete: first 2 items, then 3, then all 5
  console.log('Current extraction:', partialData.top)
  
  // Update UI progressively, save checkpoint, or trigger downstream jobs
  if (partialData.top?.length >= 3) {
    console.log('Got enough data to start processing...')
  }
}

When to use streaming:

• Live dashboards: Show extraction progress to users
• Large schemas: 50-field objects stream field-by-field
• Pipeline architectures: Start processing first results while LLM continues
• Timeout resilience: If connection drops, you have partial data saved

Example 3: Code Generation for Production Scale

This is the pro move—use AI once, run deterministic code forever:

// Phase 1: AI-assisted code generation (one-time cost)
const { code } = await scraper.generate(page, Output.object({ schema }))

// `code` contains a Playwright script string — inspect it!
console.log('Generated script:', code)

// Phase 2: Execute generated code directly (zero LLM cost)
const rawResult = await page.evaluate(code)

// Validate with your original Zod schema for safety
const data = schema.parse(rawResult)

// data is identical structure to scraper.run() output
console.log(data.top)

The economics are staggering:

For high-volume pipelines, generate once, then deploy the script to serverless functions. You get AI accuracy with traditional scraper economics.

Example 4: Multi-Modal Image Extraction

When the visual layout matters more than the markup:

// For sites with heavy visuals, charts, or complex tables
const { data } = await scraper.run(page, Output.object({ schema }), {
  format: 'image',  // Sends screenshot to multi-modal LLM
})

// GPT-4o or Claude 3 can read text in images, understand spatial relationships,
// and extract data from visual tables that would break HTML parsers

Critical requirement: Your LLM must support vision. gpt-4o, claude-3-5-sonnet-20240620, and gemini-1.5-flash all work. Llama via Groq does not support images.

Advanced Usage & Best Practices

Schema Design for Maximum Accuracy

Be explicit about types. The LLM uses your Zod schema as instructions:

// BAD: Ambiguous, LLM might return strings
const bad = z.object({ price: z.string() })

// GOOD: Explicit transformation instructions
const good = z.object({
  price: z.number().describe('Price in USD, extract from "$XX.XX" format'),
  currency: z.literal('USD').default('USD')
})

Format Mode Selection Strategy

Cost Optimization Tactics

1. Start with gpt-4o-mini or gemini-1.5-flash—test accuracy before upgrading
2. Use generate() for repeated extractions from the same site structure
3. Limit context with format: 'text' — Readability.js strips 80% of noise
4. Batch multiple schemas in single page loads to amortize browser costs

Error Handling for Production

try {
  const { data } = await scraper.run(page, Output.object({ schema }), {
    format: 'html',
  })
  
  // Zod parse for runtime safety (already done internally, but explicit is safer)
  const validated = schema.parse(data)
  return validated
} catch (error) {
  if (error instanceof z.ZodError) {
    // Schema mismatch: LLM hallucinated or page structure changed
    console.error('Validation failed:', error.issues)
    // Fallback: retry with raw_html or image mode
  }
  // Handle Playwright timeouts, LLM rate limits, etc.
}

Comparison with Alternatives

When to choose llm-scraper:

• Sites change frequently and maintenance is expensive
• You need typed data, not raw HTML
• Extraction logic is complex (conditional fields, nested relationships)
• You want to prototype fast and optimize later with generate()
• Multi-modal extraction from visual layouts is required

When NOT to use it:

• Ultra-high volume where millisecond latency matters (use generated code instead)
• Simple, stable sites where CSS selectors suffice
• Budget-constrained projects with no LLM API access

FAQ

Is llm-scraper free to use?

The library itself is open-source and free under the MIT license. However, you need API access to LLM providers (OpenAI, Anthropic, etc.), which incur usage costs. For zero-cost operation, use the Ollama integration with a local Llama model.

How accurate is AI-powered extraction compared to traditional scrapers?

For stable, simple sites, traditional methods are faster and cheaper. For dynamic, complex, or frequently-changing sites, llm-scraper achieves higher accuracy with dramatically lower maintenance. The generate() function bridges both worlds.

Can I use llm-scraper with Python or other languages?

Currently TypeScript/JavaScript only. However, you can deploy it as a microservice and call via HTTP from any language, or use the generated Playwright scripts (which are JavaScript but conceptually portable).

What happens when a website blocks Playwright?

Playwright's stealth capabilities and proxy rotation can mitigate basic detection. For advanced anti-bot systems, combine with services like ScrapingBee or Bright Data. The LLM extraction still works once you get valid page content.

How do I handle pagination and multiple pages?

llm-scraper operates on single pages. Implement pagination in your Playwright logic—navigate pages in a loop, calling scraper.run() on each. For large-scale crawling, orchestrate with BullMQ or similar job queues.

Is my data sent to third-party LLM providers?

Yes, page content is sent to your configured LLM API. For sensitive data: use Ollama for fully local processing, or ensure your provider's data handling meets compliance requirements (OpenAI's API data is not used for training as of 2024).

Can I extract from PDFs or documents, not just web pages?

Not directly—llm-scraper is web-focused. For PDFs, convert to HTML first (pdf2htmlEX, pdf.js), or use the image mode on rendered PDFs. For native PDF extraction, consider dedicated tools like unstructured.io.

Conclusion: The Future of Web Data Extraction Is Here

Let's be honest: web scraping has been a necessary evil for too long. We've accepted fragile selectors, endless maintenance, and 3 AM pages as "just part of the job." But they don't have to be.

llm-scraper represents a fundamental shift—from instructing computers how to navigate markup, to telling intelligent systems what we want. The Zod schemas you write are self-documenting contracts that both humans and AI understand. The Playwright foundation ensures you handle the modern web's complexity. And the generate() feature proves this isn't just about replacing scrapers—it's about evolving them.

Is it perfect? No. LLM costs add up at massive scale. Latency is higher than CSS selectors. And you'll still need to handle browser automation edge cases.

But for the 80% of scraping tasks that drain developer time on maintenance? This changes everything.

My recommendation: Install it today. Try the HackerNews example. Feel the rush of deleting a 200-line selector file. Then imagine what your team builds when scraping stops being a bottleneck.

Go star the repository, try it on your toughest extraction challenge, and join the community building the future of structured data. The web is messy. Your data pipeline doesn't have to be.

⭐ Star llm-scraper on GitHub ⭐