Stop Guessing Why Your Code Is Slow: Use perf-ninja Instead

Stop Guessing Why Your Code Is Slow: Use perf-ninja Instead

Your profiler lied to you. That innocent-looking loop? It's murdering your performance with cache misses. That "clean" branch you wrote? Your CPU is screaming through mispredictions while your competitors' code runs circles around yours. Here's the brutal truth: most developers have zero clue how their code actually executes on silicon. They stare at Big O complexity, pat themselves on the back, and wonder why their "optimal" algorithm gets demolished in production.

What if you could see through the matrix? What if you could decode the secret language of branch predictors, instruction caches, and execution ports? perf-ninja is that decoder ring—and it's the hands-on training ground that transforms guessing developers into performance engineers who command every CPU cycle.

Created by Denis Bakhvalov (@dendibakh), a veteran performance engineer and author of Performance Analysis and Tuning on Modern CPUs, this isn't another passive video course where you nod along and forget everything. This is deliberate practice with real code, real benchmarks, and real hardware feedback. You'll spend 90% of your time analyzing, optimizing, and proving your improvements through automated CI validation. No fluff. No theory dumps. Just pure, applied performance engineering.

Ready to stop being performance-blind? Let's dissect what makes perf-ninja the secret weapon that top-tier systems engineers are quietly using to leave competition in the dust.

What is perf-ninja?

perf-ninja is a free, open-source online course structured as hands-on laboratory assignments where developers learn to identify and eliminate low-level performance bottlenecks. Hosted at github.com/dendibakh/perf-ninja, it represents a radical departure from traditional performance education—which typically drowns learners in abstract concepts without concrete application.

Denis Bakhvalov launched this project after years of consulting on performance-critical systems and recognizing a systemic gap: engineers could recite cache hierarchy facts but couldn't translate that knowledge into measurable speedups. His book laid the theoretical foundation; perf-ninja builds the muscle memory.

The course operates on a brilliantly simple premise: you can't learn performance optimization by reading about it. Each lab presents deliberately suboptimal C++ code targeting specific microarchitectural phenomena—CPU cache misses, branch mispredictions, dependency chains, vectorization failures. Your mission? Analyze with proper tools (perf, VTune, uarch-bench), implement targeted optimizations, and submit solutions for automated benchmarking across diverse hardware platforms.

What makes perf-ninja genuinely trend-worthy is its multi-platform CI infrastructure. Your optimizations get validated on Intel Alder Lake, AMD Zen 3, and Apple M1 silicon simultaneously. This exposes a critical reality that textbook optimization ignores: performance characteristics are hardware-specific. An optimization that screams on Intel might whimper on ARM, and vice versa. This platform diversity forces you to think beyond single-architecture tuning.

The project has attracted significant community momentum with ports to Rust (perf-ninja-rs) and Zig (perf-ninja-zig), expanding its reach beyond C++ specialists. With active YouTube video companions and a growing sponsor ecosystem, perf-ninja is rapidly becoming the de facto standard for practical performance engineering education.

Key Features That Separate perf-ninja From Everything Else

Laboratory-Style Learning Architecture Unlike courses that front-load theory, perf-ninja drops you into code immediately. Each lab is a self-contained puzzle with a specific performance pathology. You'll spend your time with perf stat, perf record, and microarchitectural event counters—not watching lectures.

Automated Benchmarking & Verification Submit your optimized solution via GitHub, and the CI pipeline automatically benchmarks against baseline performance. No cheating with broken measurements. No wondering "did I actually improve anything?" The numbers don't lie, and the build badges prove it.

Multi-Architecture Hardware Validation This is where perf-ninja gets seriously impressive. The repository runs continuous integration across:

• Intel 12th-gen Alder Lake (hybrid P+E cores)
• AMD Zen 3 (aggressive branch prediction, large caches)
• Apple M1 (ARM-based, completely different memory model)

Your optimization must survive this gauntlet. This teaches portable performance thinking—the mark of senior engineers.

Structured Curriculum Across Performance Domains The labs map directly to Intel's Top-Down Microarchitecture Analysis methodology, covering:

Companion Video Ecosystem Each lab has YouTube walkthroughs where Denis demonstrates analysis techniques. The warmup video alone saves hours of tooling setup frustration.

Community & Contribution Model With multiple lab authors, active sponsors, and clear contribution guidelines, this isn't a solo project—it's a growing knowledge commons.

Real-World Scenarios Where perf-ninja Skills Dominate

Scenario 1: High-Frequency Trading Systems

In HFT, microseconds determine profitability. A branch misprediction costs 15-20 cycles—eternity when you're competing for nanosecond-level advantages. The Branches to CMOVs and Conditional Store labs teach you to eliminate unpredictable branches entirely, converting control dependencies into data dependencies that CPUs execute speculatively without penalty.

Scenario 2: Game Engine Development

Modern game engines stream massive asset datasets. The Loop Interchange, Loop Tiling, and Data Packing labs directly address the memory access patterns that separate 60fps from stuttering. Learn to transform cache-thrashing nested loops into cache-friendly traversal orders that maximize line utilization.

Scenario 3: Machine Learning Inference Pipelines

When deploying transformer models at scale, every matrix multiplication kernel matters. The Vectorization and Compiler Intrinsics labs build the skills to manually vectorize hot paths when auto-vectorization fails—critical for operations that compilers can't prove are safe to SIMD-ify.

Scenario 4: Database Query Execution

Query planners generate code that stresses every CPU subsystem. The False Sharing lab prevents catastrophic performance cliffs when multiple threads modify adjacent cache lines. The Huge Pages lab teaches TLB optimization for large working sets—essential for in-memory databases.

Scenario 5: Scientific Computing & Simulation

Weather prediction, fluid dynamics, molecular dynamics—all depend on stencil computations with predictable memory patterns. The SW Memory Prefetching and Memory Alignment labs enable software-controlled prefetching that hides latency for access patterns hardware prefetchers can't predict.

Step-by-Step Installation & Setup Guide

Getting started with perf-ninja requires proper toolchain configuration. Follow these steps precisely—skipping prerequisites leads to misleading benchmark results.

Prerequisites

Mandatory:

• Basic C++ proficiency (templates, pointers, references)
• Git for version control
• CMake 3.13+ for build generation
• A C++17-compatible compiler (GCC 9+, Clang 10+, MSVC 2019+)

Strongly Recommended:

• Linux environment (native or WSL2)—most performance analysis tools are Linux-first
• perf tool installed (linux-tools-generic package on Ubuntu)
• Denis' book Performance Analysis and Tuning on Modern CPUs for conceptual foundation

Clone and Configure

# Clone the repository with all lab assignments
git clone https://github.com/dendibakh/perf-ninja.git
cd perf-ninja

# Read the critical setup documentation FIRST
cat GetStarted.md

The GetStarted.md file contains platform-specific instructions that evolve with the project. Do not skip this.

Build System Setup

# Create dedicated build directory (out-of-source builds mandatory)
mkdir build && cd build

# Generate build files with optimizations enabled for benchmarking
cmake .. -DCMAKE_BUILD_TYPE=Release

# Compile all lab assignments (or target specific ones)
cmake --build . --parallel $(nproc)

Critical configuration note: The build system uses Google Benchmark for measurements. Release mode is essential—Debug builds disable optimizations and distort results completely.

Verification

# Run a specific lab's benchmark to verify setup
./labs/misc/warmup/warmup_bench

# Expected output: baseline performance numbers with statistical confidence intervals

Tooling Installation (Linux)

# Install perf and related tools for hardware counter access
sudo apt-get update
sudo apt-get install linux-tools-common linux-tools-generic linux-tools-$(uname -r)

# Verify perf can access CPU events
perf stat -e cycles,instructions,cache-misses ls

IDE Integration

For effective analysis, configure your editor to:

1. Display assembly alongside source (VS Code: "C/C++: Disassembly View")
2. Highlight compiler warnings at maximum level (-Wall -Wextra)
3. Integrate with clangd for precise code navigation

REAL Code Examples: Inside the Labs

Here are actual patterns from perf-ninja labs, dissected to reveal the optimization opportunities.

Example 1: Warmup Lab — Baseline Measurement Discipline

The warmup lab teaches the most underrated performance skill: measuring before optimizing. Here's the conceptual pattern:

// labs/misc/warmup/solution.cpp
// This lab establishes benchmark discipline: stable measurement environment,
// statistical significance, and understanding variance sources.

#include <benchmark/benchmark.h>
#include <vector>
#include <random>

// Volatile prevents complete dead-code elimination
// while allowing the compiler to optimize the computation itself
static volatile int sink;

void benchmark_sum(benchmark::State& state) {
    // Setup: generate test data once, outside measurement loop
    std::vector<int> data(100000);
    std::mt19937 gen(42);  // Fixed seed for reproducibility
    std::uniform_int_distribution<> dis(1, 100);
    for (auto& x : data) x = dis(gen);
    
    // Measurement loop: Google Benchmark automatically determines iterations
    for (auto _ : state) {
        int sum = 0;
        // Benchmark-friendly pattern: accumulate to local, write to volatile sink after
        for (int x : data) {
            sum += x;  // Compiler can vectorize this loop
        }
        sink = sum;  // Force side effect, prevent optimization away
    }
    
    // Report throughput: bytes processed per second
    state.SetBytesProcessed(state.iterations() * data.size() * sizeof(int));
}

BENCHMARK(benchmark_sum);
BENCHMARK_MAIN();

Why this matters: The sink pattern is crucial—without it, compilers eliminate "useless" computations and you benchmark nothing. The SetBytesProcessed call enables normalized comparison across different input sizes. This discipline separates legitimate optimizations from measurement artifacts.

Example 2: Vectorization Lab — SIMD Transformation

The vectorization_1 lab confronts you with code that compilers fail to auto-vectorize. Here's a representative optimization pattern:

// BEFORE: Scalar loop, compiler can't prove safety
void sum_arrays_scalar(const float* __restrict a,
                       const float* __restrict b,
                       float* __restrict c,
                       size_t n) {
    for (size_t i = 0; i < n; ++i) {
        c[i] = a[i] + b[i];  // May vectorize, but aliasing fears prevent it
    }
}

// AFTER: Explicit vectorization with compiler intrinsics
#include <immintrin.h>  // AVX2 intrinsics header

void sum_arrays_avx2(const float* __restrict a,
                     const float* __restrict b,
                     float* __restrict c,
                     size_t n) {
    // Process 8 floats per iteration (256-bit AVX2 register / 32-bit float)
    size_t i = 0;
    for (; i + 8 <= n; i += 8) {
        // Load 8 floats from each array into SIMD registers
        __m256 va = _mm256_loadu_ps(&a[i]);  // Unaligned load (safe, fast on modern CPUs)
        __m256 vb = _mm256_loadu_ps(&b[i]);
        
        // Single instruction, 8 additions in parallel
        __m256 vc = _mm256_add_ps(va, vb);
        
        // Store result back
        _mm256_storeu_ps(&c[i], vc);
    }
    
    // Scalar cleanup for remaining elements (n % 8)
    for (; i < n; ++i) {
        c[i] = a[i] + b[i];
    }
}

The insight: __restrict promises no aliasing, but compilers remain conservative. Manual intrinsics guarantee vectorization. The _mm256_loadu_ps (unaligned load) is intentionally used—modern Intel/AMD CPUs have negligible penalty for unaligned 256-bit loads, eliminating alignment complexity. The cleanup loop handles the "tail" when n isn't divisible by vector width.

Example 3: False Sharing Lab — Cache Line Contention

The false_sharing_1 lab reveals how "innocent" multithreaded code becomes 100x slower:

// BEFORE: Catastrophic false sharing
struct Counter {
    uint64_t value = 0;  // 8 bytes, but cache lines are 64 bytes
};

// Counters likely placed adjacent by allocator → same cache line
std::array<Counter, 8> counters;

void increment_bad(std::array<Counter, 8>& counters, size_t id, size_t iterations) {
    for (size_t i = 0; i < iterations; ++i) {
        counters[id].value++;  // Each thread invalidates others' cache!
    }
}
// 8 threads, each writing "independent" counter → 8x cache line bouncing

// AFTER: Cache line padding ensures separation
struct alignas(64) PaddedCounter {  // Force 64-byte alignment = cache line size
    uint64_t value = 0;
    // Implicit padding: 56 bytes to fill cache line (64 - 8 = 56)
    // Or explicit: char padding[56];
};

std::array<PaddedCounter, 8> padded_counters;

void increment_good(std::array<PaddedCounter, 8>& counters, size_t id, size_t iterations) {
    for (size_t i = 0; i < iterations; ++i) {
        counters[id].value++;  // Each counter on unique cache line
    }
}
// Zero cross-thread cache invalidation → linear scaling

The brutal truth: alignas(64) is C++11's most underutilized performance feature. Without it, your "lock-free" algorithms may perform worse than single-threaded code. The perf c2c tool (cache-to-cache analysis) can detect this in production—perf-ninja teaches you to prevent it during design.

Example 4: Branch Optimization — Converting to Data Dependencies

From the branches_to_cmov_1 lab, eliminating unpredictable branches:

// BEFORE: Unpredictable branch, pipeline flushes on misprediction
int max_branch(int a, int b) {
    if (a > b)  // Branch predictor struggles with random data
        return a;
    else
        return b;
}

// AFTER: Conditional move, no branch prediction needed
#include <cstdint>

int max_cmov(int a, int b) {
    // Compiler emits CMOVcc instruction: data-dependent, always predicted correctly
    // Equivalent to: return (a > b) ? a : b; with optimization enabled
    return a ^ ((a ^ b) & -(a < b));  // Branchless, bit-twiddling version
}

// Modern compiler with -O2 or higher automatically generates CMOV
// The explicit bit version demonstrates the principle for complex cases

When this matters: Branch prediction fails with data-dependent control flow (hash table lookups, parsing irregular formats). CMOV converts control hazard to data hazard—which modern CPUs handle with speculative execution. The bit-twiddling version is educational; in practice, write clean ?: and trust the compiler.

Advanced Usage & Best Practices

Hardware Counter Deep Dives Don't stop at cycles and instructions. perf-ninja rewards mastery of specific events:

• L1-dcache-load-misses → Memory bound classification
• branch-misses → Bad speculation quantification
• uops_issued.any / uops_retired.retire_slots → Frontend vs backend bound

Cross-Platform Validation Strategy After optimizing, always verify on at least two architectures. An optimization exploiting Intel's 512-bit AVX-512 may throttle on AMD. Apple's M1 has no AVX at all—NEON intrinsics required for SIMD there.

Compiler Flag Hygiene

# Baseline: optimize but don't cheat
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer"

# The frame pointer enables accurate profiling with perf record -g
# Slight register pressure cost, massive debuggability gain

Statistical Rigor Run benchmarks multiple times, check variance. Google Benchmark reports automatically, but for perf stat, use:

perf stat -r 10 -e cycles,instructions ./your_benchmark

Reading Assembly Without Fear objdump -d -M intel your_binary | less is your friend. Learn to recognize:

• Vector instructions (vmovups, vaddps, vfma...) for successful SIMD
• Conditional jumps (jne, jg) where CMOV should exist
• Memory operands with complex addressing ([rax + rdx*8 + 16]) for indexing costs

Comparison With Alternatives

The verdict: perf-ninja occupies a unique position—more rigorous than video courses, more practical than books, more accessible than university programs, and completely free. The automated benchmarking infrastructure alone eliminates the "works on my machine" syndrome that plagues self-directed learning.

FAQ: What Developers Ask About perf-ninja

Q: Is perf-ninja suitable for beginners in C++? A: No. The prerequisites explicitly require basic C++ fluency. If you're still learning move semantics or template basics, build that foundation first. The labs assume you can read and modify non-trivial C++ without hand-holding.

Q: Can I use Windows or macOS exclusively? A: Technically yes—the CI validates on all platforms. Practically, Linux is strongly recommended. The perf tool ecosystem is Linux-native, and most analysis techniques translate poorly to Windows ETW or macOS Instruments. WSL2 on Windows is an excellent compromise.

Q: How long does each lab take? A: Estimates range from 30 minutes to 4 hours depending on your background and the complexity. Vectorization labs may click immediately for SIMD-experienced developers; memory optimization labs often surprise even senior engineers with subtle layout effects.

Q: Do I need Denis' book to succeed? A: It's listed as "recommended," not required. The book provides conceptual scaffolding for why techniques work; the labs provide muscle memory for applying them. If you're struggling with terminology (Top-Down analysis, execution ports, retirement), the book will accelerate comprehension significantly.

Q: Can I contribute new labs? A: Absolutely! The project welcomes contributions. See Contributing.md for guidelines. Multiple lab authors are already credited—this is genuinely community-driven expansion.

Q: Will learning perf-ninja help my career? A: Performance engineering is among the highest-compensated specialization tracks in systems programming. Companies building databases, game engines, financial systems, and cloud infrastructure desperately need engineers who understand silicon-level behavior. This skillset differentiates you from the masses writing framework-dependent code.

Q: What's the difference between perf-ninja and just reading Agner Fog's manuals? A: Agner's manuals are the definitive reference—encyclopedic, exhaustive, and demanding. perf-ninja is the deliberate practice that translates reference knowledge into applied skill. Use both: manuals for deep understanding, labs for verified capability.

Conclusion: Your CPU Deserves Better

Stop accepting mysterious performance regressions. Stop blaming "the compiler" or "the framework" when your code executes poorly. The hardware gives you precise feedback—you just need to learn its language.

perf-ninja is that Rosetta Stone. Through 20+ hands-on laboratories, multi-platform automated validation, and a community of engineers who refuse to tolerate inefficiency, it transforms performance from black magic into engineering discipline. The skills you build here—reading hardware counters, reasoning about cache geometry, eliminating speculative execution hazards—pay dividends across every system you'll ever build.

Denis Bakhvalov has done something remarkable: he's made elite performance engineering accessible without diluting its rigor. The course is free. The hardware validation is real. The community is growing.

Your move. Clone github.com/dendibakh/perf-ninja, read GetStarted.md, watch the warmup video, and run your first benchmark. That sluggish code you've been ignoring? It's about to reveal all its secrets. Welcome to the elite tier of systems performance engineers.

Ready to optimize? Star the repository, sponsor the project if it accelerates your growth, and start with the warmup lab today.