Developer Tools Machine Learning 66 vues

Train GPT-2 for $48? Karpathy's nanochat Exposes the Secret

M
Mohamed IDBRAHIM
Auteur
Train GPT-2 for $48? Karpathy's nanochat Exposes the Secret

Train GPT-2 for $48? Karpathy's nanochat Exposes the Secret

What if I told you that a capability once reserved for well-funded AI labs with million-dollar budgets is now sitting in your pocket change? In 2019, OpenAI spent approximately $43,000 training GPT-2—a model that stunned the world with its text generation abilities. Fast forward to today, and Andrej Karpathy just dropped a bomb on the entire machine learning community: you can now achieve the exact same capability for roughly $48. Not $4,800. Not $480. Forty-eight dollars. On a single GPU node. In about two hours.

Stop and absorb that for a second.

The democratization of artificial intelligence isn't coming—it's already here, and it's called nanochat. This isn't another bloated framework drowning in configuration files and abstracted layers you can't debug. This is Karpathy's raw, minimal, deliberately hackable experimental harness for training large language models from scratch. If you've ever felt locked out of the LLM revolution because you don't have Big Tech's compute budget, nanochat isn't just a tool. It's your key.

But here's the real kicker most people miss: nanochat isn't merely about cost reduction. It's about cognitive accessibility. The entire codebase is designed around a single complexity dial—--depth—that automatically configures everything else. No more guessing learning rates, hunting for optimal batch sizes, or drowning in hyperparameter search grids. You want a bigger model? Turn the dial. You want smaller? Turn it back. Everything else just works.

Ready to understand how Karpathy pulled off this insane feat? Let's dive deep into what makes nanochat the most exciting open-source LLM project of 2025—and why you should be forking it right now.

What is nanochat?

nanochat is the simplest experimental harness for training large language models, created by Andrej Karpathy—yes, the former Tesla AI director, OpenAI founding member, and the educator behind some of the most beloved deep learning courses on the internet. The repository lives at github.com/karpathy/nanochat and carries the audacious tagline: "The best ChatGPT that $100 can buy."

But nanochat is far more than a budget training script. It's a complete end-to-end pipeline covering every major stage of modern LLM development: tokenization, pretraining, supervised fine-tuning (SFT), reinforcement learning (RL), evaluation, inference, and even a polished ChatGPT-like web UI for chatting with your freshly trained model.

The project draws direct lineage from Karpathy's earlier nanoGPT (pretraining only) and takes heavy inspiration from modded-nanoGPT by Keller Jordan, which gamified LLM training with leaderboards and clear metrics. What nanochat adds is the full conversational stack—transforming a pretrained base model into something you can actually talk to.

Why is it trending now? Three explosive factors converged:

  1. Compute cost collapse: Cloud GPU prices have plummeted, with spot instances making 8xH100 nodes accessible to individual researchers and hobbyists.
  2. Algorithmic efficiency gains: Seven years of advances across the entire stack—from optimized optimizers like Muon to better data curation (NVIDIA ClimbMix, Fineweb)—have compressed what required weeks into hours.
  3. The "GPT-2 speedrun" movement: nanochat's leaderboard has turned LLM training into a competitive sport, with researchers racing to beat OpenAI's 2019 benchmark faster and cheaper.

The current record? 1.65 hours to surpass GPT-2's DCLM CORE score of 0.256525. That's not a typo. What once consumed 168 hours of training (OpenAI's original run) now completes before your coffee goes cold.

Key Features That Make nanochat Insane

Karpathy didn't just build another training framework—he engineered a philosophy of radical simplicity. Here's what separates nanochat from the ocean of LLM tools:

Single-Dial Complexity Control

The entire hyperparameter space collapses into one integer: --depth, the number of transformer layers. Set depth=12 for quick experiments (~GPT-1 scale). Crank it to depth=26 for GPT-2 capability. The width, head count, learning rate, training horizon, weight decay—all compute-optimal values are automatically derived. No more configuration hell. No more grid searches. This is machine learning as it should be: you control the model size, the math handles the rest.

Explicit Precision Management

Forget torch.amp.autocast and its black-box behavior. nanochat uses a single global COMPUTE_DTYPE with hardware-aware defaults:

Hardware Default dtype Rationale
CUDA SM 80+ (A100, H100) bfloat16 Native bf16 tensor cores
CUDA SM < 80 (V100, T4) float32 No bf16 support; fp16 available via env override
CPU / Apple Silicon float32 No reduced-precision tensor cores

Model weights stay in fp32 for optimizer precision, but a custom Linear layer casts to compute dtype during forward passes. Embeddings store directly in compute dtype for memory savings. You get full explicit control with the same mixed-precision benefits—no magic, no surprises.

Complete Pipeline in One Repo

Most projects give you pieces. nanochat gives you the entire factory: train a tokenizer, pretrain a base model, evaluate with DCLM CORE metrics, run SFT for instruction following, apply RL for refinement, launch inference with KV-cache optimization, and serve through a web UI. The file structure tells the story:

  • scripts/base_train.py → Pretrain your foundation model
  • scripts/chat_sft.py → Supervised fine-tuning for conversational ability
  • scripts/chat_rl.py → Reinforcement learning optimization
  • scripts/chat_web.py → Launch the ChatGPT-like interface
  • scripts/chat_cli.py → Quick terminal conversations

Competitive Speedrun Leaderboard

The "Time-to-GPT-2" leaderboard drives relentless optimization. Current standings show progression from 3.04 hours down to 1.65 hours, with innovations like FP8 training, million-token batch sizes, and autoresearched hyperparameters. Every improvement is validated across all depth settings—no fragile, one-off tricks allowed.

Hardware Flexibility

Designed for 8xH100 nodes but gracefully degrades: single GPU works (8x slower), A100 nodes function (slightly slower), CPU/MPS runs for experimentation (dramatically shrunk models). VRAM constraints? Tune --device-batch-size down from 32 to 1. The code adapts automatically to gradient accumulation when torchrun is omitted.

Real-World Use Cases Where nanochat Dominates

1. Research Prototyping at Ludicrous Speed

Academic researchers and indie labs can iterate on architectural hypotheses in minutes instead of weeks. The d12 (12-layer) configuration trains in approximately 5 minutes, perfect for ablation studies. Monitor val_bpb, core_metric, train/mfu, and tok_per_sec on Weights & Biases to instantly evaluate if your modification helps. No more burning thousands in cloud credits to discover your idea doesn't work.

2. Custom Domain Chatbots on a Shoestring

Need a specialized conversational AI for healthcare, legal, or technical domains? Pretrain on domain data, run SFT with synthetic instruction data, and deploy. The identity infusion guide shows how to generate synthetic personality data and blend it into training. Total cost: under $100 for a model you fully own and control.

3. AI Education and Reproducible Science

nanochat is the pedagogical tool for understanding modern LLMs. Every component is readable, hackable, and traceable. Students can follow data from raw text → tokens → embeddings → transformer layers → probability distributions → generated text. No opaque abstractions. The codebase is your textbook.

4. Edge and Privacy-Critical Deployments

Train entirely on-premise or in your VPC. No API calls to external services. No data exfiltration risks. For organizations handling sensitive information—healthcare providers, financial institutions, defense contractors—nanochat enables sovereign AI without the traditional multi-million dollar infrastructure investment.

Step-by-Step Installation & Setup Guide

Getting nanochat running is deliberately frictionless. Here's the complete path from zero to chatting with your own LLM:

Prerequisites

  • A GPU node (8xH100 recommended for speedrun, any CUDA GPU works)
  • uv installed (docs.astral.sh/uv)
  • Basic familiarity with Python↗ Bright Coding Blog and terminal navigation

Installation Commands

# Clone the repository
git clone https://github.com/karpathy/nanochat.git
cd nanochat

# Install dependencies with CUDA support
uv sync --extra gpu

# Alternative: CPU-only or Apple Silicon
# uv sync --extra cpu

# Activate the virtual environment
source .venv/bin/activate

# For development (adds pytest, matplotlib, transformers, etc.)
# uv sync --extra gpu --group dev

Reproduce GPT-2 and Launch the Chat UI

The legendary speedrun.sh script encapsulates the entire pipeline:

# Start a screen session (training takes ~3 hours)
screen -S nanochat-speedrun

# Launch the full training pipeline
bash runs/speedrun.sh

This single script handles: data download, tokenization, distributed pretraining across 8 GPUs, evaluation, SFT, and checkpoint saving. The current reference implementation trains a depth-24 to depth-26 model to GPT-2 capability.

Once training completes, serve your model:

# Ensure virtual environment is active
source .venv/bin/activate

# Launch the ChatGPT-like web interface
python -m scripts.chat_web

Visit the displayed URL (e.g., http://YOUR_NODE_IP:8000/ on Lambda Cloud) and start conversing. Ask it philosophical questions, request creative writing, or probe its hallucinations—it's genuinely surreal to interact with something you birthed from random initialization two hours prior.

Running on Smaller Hardware

# Single GPU (8x slower, identical results)
python -m scripts.base_train --depth=12 --device-batch-size=4

# CPU / MPS experimentation (dramatically reduced model)
bash runs/runcpu.sh

Quick Research Iteration Loop

# 5-minute d12 experiment for testing modifications
OMP_NUM_THREADS=1 torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=12 \
    --run="d12" \
    --model-tag="d12" \
    --core-metric-every=999999 \
    --sample-every=-1 \
    --save-every=-1

Monitor on Weights & Biases, iterate on your code change, rerun. This tight feedback loop is where nanochat's true power shines.

REAL Code Examples from nanochat

Let's examine actual code from the repository to understand how Karpathy achieves this elegant simplicity.

Example 1: The Single-Dial Training Launch

This is the core research iteration command, taken directly from the README:

# Launch distributed training with automatic hyperparameter derivation
OMP_NUM_THREADS=1 torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=12 \              # SINGLE DIAL: 12 transformer layers
    --run="d12" \             # Weights & Biases run identifier
    --model-tag="d12" \       # Checkpoint naming prefix
    --core-metric-every=999999 \  # Only compute expensive CORE eval at end
    --sample-every=-1 \       # Disable intermediate text generation samples
    --save-every=-1           # Disable intermediate checkpoint saves

What's happening here? The magic is --depth=12. Behind the scenes, nanochat's configuration system automatically calculates:

  • Model width (embedding dimension scales with depth)
  • Number of attention heads (maintaining head dimension consistency)
  • Learning rate (adjusted via scaling laws)
  • Training steps and batch sizes (compute-optimal training horizon)
  • Weight decay and other regularization parameters

The OMP_NUM_THREADS=1 prevents OpenMP from fighting PyTorch's own threading. torchrun with --nproc_per_node=8 distributes across 8 GPUs using PyTorch's native DDP. The negative values for sampling and saving streamline the experiment—when you're iterating rapidly, you only care about final metrics.

Example 2: Explicit Precision Control

nanochat rejects PyTorch's autocast in favor of transparent dtype management:

# Force fp32 for debugging numerical issues
NANOCHAT_DTYPE=float32 python -m scripts.chat_cli -p "hello"

# Force bfloat16 on supported hardware (explicit override)
NANOCHAT_DTYPE=bfloat16 torchrun --nproc_per_node=8 -m scripts.base_train

Why this matters: torch.amp.autocast implicitly selects operations to run in reduced precision, which can create impossible-to-debug numerical instabilities. nanochat's approach stores optimizer state in full fp32 (critical for training stability), casts weights to COMPUTE_DTYPE only during forward passes via a custom Linear layer, and keeps embeddings in reduced precision for memory efficiency. You know exactly what precision every operation uses because you set it explicitly.

The environment variable override means you can debug in fp32, then deploy in bf16, without code changes. This is engineering for reproducibility.

Example 3: The Complete Speedrun Pipeline

The speedrun.sh script that trains GPT-2 for ~$48:

bash runs/speedrun.sh

While the full script isn't shown in the README, its existence as a single file encapsulates nanochat's philosophy. Compare this to other frameworks requiring:

  • Separate data preprocessing scripts
  • Manual distributed launch configurations
  • Post-training conversion utilities
  • External serving infrastructure setup

nanochat's speedrun is: run one script, wait ~3 hours, talk to your model. The complexity isn't hidden—it's eliminated through thoughtful design.

Example 4: Web UI Launch

python -m scripts.chat_web

This launches a complete ChatGPT-like interface using the HTML/CSS/JS in nanochat/ui.html. No separate frontend build step. No npm install. No React↗ Bright Coding Blog dependencies. A self-contained, zero-dependency web interface served directly from Python. Access it via your node's public IP and port 8000.

The simplicity is almost aggressive. And that's precisely the point.

Advanced Usage & Best Practices

Monitoring Training Like a Pro

Track these Weights & Biases metrics for optimization insights:

  • val_bpb (bits per byte): Vocab-size-invariant validation loss. Lower is better. Compare across different tokenizers fairly.
  • core_metric: DCLM CORE score—the gold standard for GPT-2 capability comparison. Target: exceed 0.256525.
  • train/mfu: Model FLOPS utilization. Percentage of theoretical peak GPU performance. Low values indicate inefficient training—debug your data loading or communication overhead.
  • train/tok_per_sec: Raw throughput. Useful for hardware comparisons and spotting regressions.
  • VRAM utilization: Watch for memory pressure that could trigger costly CUDA OOM kills.

The Depth Sweep Strategy

To generate the nanochat "miniseries" of compute-optimal models:

for depth in 8 12 16 20 24 26 30; do
    torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
        --depth=$depth --run="d${depth}" --model-tag="d${depth}"
done

Each model is automatically configured for optimal training. Plot the Pareto frontier of capability versus cost.

Contributing Improvements

nanochat's governance is strict: changes must work across all depth settings. No fragile optimizations that only help at one scale. Before submitting PRs, validate on d12 (fast) and d24-d26 (target). Disclose any LLM-assisted code you don't fully understand—transparency maintains the codebase's educational value.

Comparison with Alternatives

Feature nanochat Hugging Face Transformers LLaMA-Factory DeepSpeed
Lines of Code ~2,000 (minimal) ~500,000+ (framework) ~50,000+ (framework) ~100,000+ (framework)
Single GPU Node ✅ Native design ⚠️ Possible but complex ⚠️ Possible but complex ❌ Designed for clusters
End-to-End Pipeline ✅ Complete (token → chat) ❌ Requires external tools ⚠️ Partial ❌ Training only
Hyperparameter Automation ✅ Single --depth dial ❌ Manual configuration ⚠️ Some presets ❌ Manual configuration
Hackability ✅ Designed for forks ❌ Abstracted internals ⚠️ Moderate ❌ Complex internals
Cost to GPT-2 ~$48 $500+ (expert setup) $300+ $200+ (infrastructure)
Educational Value ✅ Maximum transparency ⚠️ High-level APIs hide details ⚠️ Moderate ❌ Infrastructure-focused
Production Scale ❌ Research/experimental ✅ Enterprise ready ✅ Fine-tuning focus ✅ Massive scale

Choose nanochat when: You're learning, researching, prototyping, or need full ownership of a small-to-medium model. You value code readability over feature breadth. Your budget is measured in hundreds, not millions.

Choose alternatives when: You need production-grade serving at billion-parameter scale, enterprise support, or seamless integration with existing MLOps infrastructure.

FAQ

Q: Can I really train GPT-2 for under $50? A: Yes. The reference speedrun costs ~$48 on an 8xH100 node at ~$3/GPU/hour for ~2 hours. Spot instances drop this to ~$15. This isn't theoretical—it's the documented, reproducible benchmark.

Q: What GPU do I actually need? A: 8xH100 (80GB) is optimal for the speedrun. 8xA100 works slightly slower. Single GPU of any size works with 8x time penalty and --device-batch-size tuning. Sub-80GB GPUs require batch size reduction.

Q: How does nanochat differ from nanoGPT? A: nanoGPT was pretraining-only. nanochat adds the complete conversational stack: SFT, RL, evaluation, inference optimization, and a web UI. It's the full journey from data to chatbot.

Q: Is the trained model commercially usable? A: nanochat itself is MIT licensed. Model weights you train depend on your training data's licenses. Pretraining on public data (Fineweb, ClimbMix) and your own synthetic data generally permits commercial use, but verify specific dataset licenses.

Q: Can I run this on my laptop? A: Technically yes via runcpu.sh, but results will be weak. The d12 configuration trains in ~5 minutes on 8xH100; on CPU it's dramatically longer with a shrunk model. For serious results, cloud GPUs are essential.

Q: What's the "GPT-2 speedrun" and why does it matter? A: It's a community competition to train GPT-2-capable models faster and cheaper. The leaderboard tracks wall-clock time to exceed OpenAI's 2019 CORE score. It matters because it validates that AI capability is becoming accessible, not concentrated.

Q: How do I add custom abilities to my nanochat? A: See the counting 'r' in strawberry guide. Generate synthetic training data for your target skill, blend it into SFT data, and retrain. The pattern works for any discrete capability.

Conclusion

nanochat isn't just a repository—it's a declaration of independence for individual AI researchers, hackers, and curious developers worldwide. Andrej Karpathy has distilled the entire modern LLM pipeline into something you can read in an afternoon, modify in an evening, and train over a lunch break. The $48 GPT-2 isn't a stunt; it's proof that seven years of algorithmic progress, when packaged with intentional simplicity, can collapse economic barriers that once seemed insurmountable.

But the deeper value? Understanding. In an era where AI feels increasingly like magic performed by distant corporations, nanochat hands you the wand and shows you every spell. You see the tokens flow, the gradients propagate, the weights update. You talk to something you created from nothing.

The leaderboard will keep falling. The community will keep optimizing. And somewhere, someone reading this will fork nanochat, make a breakthrough, and push the frontier further.

That someone could be you.

Fork nanochat on GitHub today. Spin up a GPU node. Run the speedrun. Talk to your creation. Welcome to the future of accessible AI—it's already here, it's already cheap, and it's waiting for you to hack on it.

Commentaires 0

Aucun commentaire pour l'instant. Soyez le premier à réagir !

Laisser un commentaire