TriAttention: 10.7x KV Compression That Actually Works

Your GPU is screaming. That 32B parameter reasoning model you desperately want to run locally? It's demanding 48GB+ of VRAM just for the KV cache alone. You've tried quantization. You've tried pruning. You've even considered selling your kidney for an H100. But what if I told you there's a method that compresses KV cache by 10.7x—while actually matching full attention accuracy—and lets you deploy serious reasoning models on a humble 24GB RTX 4090?

Welcome to TriAttention, the trigonometric compression technique that's making AI researchers at MIT and NVIDIA rethink everything about efficient inference. This isn't another half-baked approximation that trades accuracy for speed. This is frequency-domain sorcery that understands how attention really behaves in long reasoning chains. And it's about to change how you deploy large language models forever.

What Is TriAttention?

TriAttention is an open-source KV cache compression system developed by researchers from MIT, NVIDIA, and Zhejiang University (ZJU). Born from a critical observation about how attention patterns behave in modern reasoning models, it represents a fundamental departure from existing compression methods like SnapKV and R-KV.

The project emerged from a deceptively simple insight: pre-RoPE Q/K vectors in long reasoning models concentrate around fixed centers. These centers determine distance preferences through a trigonometric series—a mathematical property that existing compression methods completely ignore. Instead of selecting representative queries and approximating attention scores, TriAttention directly scores keys using these discovered centers and norms.

This architectural difference matters enormously. Traditional methods like SnapKV rely on selecting "important" tokens based on query representations, which introduces overhead and approximation errors. TriAttention bypasses this entirely by leveraging the inherent trigonometric structure of attention in reasoning models. The result? 2.5x throughput improvement on AIME25 long reasoning tasks with zero accuracy degradation—40.8% accuracy versus 40.8% for full attention.

The repository has exploded in popularity since its release, with rapid community adoption including ports to SGLang, llama.cpp, MLX for Apple Silicon, and even DGX Spark (GB10) enablement. The project's Apache 2.0 license and active maintenance make it production-ready for serious deployments.

Key Features That Separate TriAttention from the Pack

Trigonometric Frequency-Domain Compression The core innovation: instead of operating in the token space like competitors, TriAttention exploits the frequency-domain structure of attention. Pre-RoPE query and key vectors cluster around specific centers that follow predictable trigonometric patterns. By scoring keys directly through these centers, the method achieves compression without the proxy selection overhead that plagues alternatives.

Massive Memory Reduction Without Compromise

• 10.7x KV memory reduction with calibrated statistics
• 2.5x throughput boost on AIME25 reasoning benchmarks
• Zero accuracy loss compared to full attention (verified: 40.8% vs 40.8%)
• 6.3x speedup on MATH-500 with only 1.2% accuracy drop (69.6% → 68.4%)

Production-Ready vLLM Integration TriAttention ships as a transparent vLLM plugin—install it, set environment variables, and your existing vLLM deployment automatically compresses KV caches. No code changes. No model retraining. No architectural modifications. The plugin patches scheduler and worker components automatically, exposing standard OpenAI-compatible APIs.

OpenClaw Local Deployment The killer feature for practitioners: TriAttention enables OpenClaw-compatible local deployment on 24GB RTX 4090 cards. Run Qwen3-32B-int4 models that previously required datacenter GPUs. The combination of INT4 quantization with trigonometric KV compression makes consumer-grade hardware viable for serious reasoning workloads.

Broad Hardware Support Beyond NVIDIA GPUs, the ecosystem now covers:

• Apple Silicon (M1/M2/M3/M4) via MLX integration
• AMD GPUs via community llama.cpp port with HIP/ROCm
• DGX Spark/GB10 (sm-121 architecture) with native vLLM support

Real-World Use Cases Where TriAttention Dominates

1. Local Reasoning Model Deployment

You're building a coding assistant or math tutor that runs entirely on-premise. Full DeepSeek-R1-Distill-Qwen-7B demands ~35GB KV cache for long reasoning chains. With TriAttention's 2048-token budget, you squeeze into under 4GB—making 24GB consumer GPUs viable for multi-turn conversations with complex chain-of-thought reasoning.

2. High-Throughput API Serving

Your startup serves thousands of concurrent users with reasoning-heavy queries. Standard vLLM chokes on KV cache allocation, forcing expensive GPU scaling. TriAttention's 6.3x throughput improvement on MATH-500 means serving the same traffic with one-sixth the infrastructure—directly translating to 83% reduction in inference costs.

3. Long-Context Video Generation

The project's AR video generation support (via LongLive integration) addresses a screaming need in creative AI. Autoregressive video models generate enormous KV caches frame-by-frame. TriAttention's compression prevents the exponential memory growth that kills long video sequences, enabling generation of minutes rather than seconds of coherent video.

4. Edge and Mobile Inference

The MLX port for Apple Silicon and community ggml implementation bring serious compression to resource-constrained environments. Run distilled reasoning models on MacBook Pros or even explore iPad deployment—territory completely inaccessible to uncompressed attention.

5. Multi-Turn Conversational AI

Chat applications accumulate context across dozens of turns. Without compression, KV cache grows linearly with conversation length, eventually forcing context truncation or OOM crashes. TriAttention's configurable budget (recommended: 12,000 tokens for chat) maintains conversation coherence while capping memory usage—critical for production assistant deployments.

Step-by-Step Installation & Setup Guide

Standard Installation (NVIDIA GPUs)

# Clone the repository
git clone https://github.com/WeianMao/triattention.git
cd triattention

# Install TriAttention in editable mode
pip install -e .

# Install Flash Attention (recommended but slow on some platforms)
pip install flash-attn --no-build-isolation
# Note: Takes ~105 minutes on DGX Spark / GB10; consider pre-built wheels if available

DGX Spark / GB10 Special Installation

For NVIDIA's compact AI workstation with sm-121 architecture, use this optimized path:

# Create uv virtual environment for faster package resolution
uv venv
. .venv/bin/activate

# Install PyTorch with CUDA 13.0 support
uv pip install --index-url https://download.pytorch.org/whl/cu130 torch torchvision torchaudio

# Install vLLM pre-built wheel for aarch64
uv pip install \
  https://github.com/vllm-project/vllm/releases/download/v0.19.0/vllm-0.19.0-cp38-abi3-manylinux_2_31_aarch64.whl \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --extra-index-url https://pypi.org/simple \
  --index-strategy unsafe-best-match

# Install TriAttention
uv pip install -e .

# Configure Triton cache for compilation artifacts
export TRITON_CACHE_DIR=~/.cache/.triton-cache
mkdir -p $TRITON_CACHE_DIR

# Set library paths for CUDA and PyTorch
PY_SITE=$(.venv/bin/python -c "import sysconfig; print(sysconfig.get_paths()['purelib'])")
export LD_LIBRARY_PATH="$PY_SITE/torch/lib:$PY_SITE/nvidia/cu13/lib:/usr/local/cuda/targets/sbsa-linux/lib:${LD_LIBRARY_PATH:-}"

Quick Benchmark Verification

# Run single evaluation to verify installation
python scripts/cli.py run-one \
    --model Qwen3-8B \
    --dataset aime24 \
    --method triattention \
    --budget 2048

Datasets (AIME 2024, AIME 2025, MATH-500) auto-download from HuggingFace on first run—zero manual preparation needed.

REAL Code Examples from the Repository

Example 1: Production vLLM Server with OpenAI-Compatible API

This is the bread-and-butter deployment pattern—transparent compression with standard API access:

# Configure compression parameters via environment variables
export TRIATTN_RUNTIME_KV_BUDGET=2048  # Maximum retained tokens per request
export TRIATTN_RUNTIME_SPARSE_STATS_PATH=triattention/vllm/stats/qwen3_32b_int4_stats.pt

# Launch vLLM server—TriAttention auto-activates, patches scheduler and worker
# Set ENABLE_TRIATTENTION=0 to disable if needed
vllm serve <model_path> \
    --dtype bfloat16 \
    --max-model-len 32768 \
    --enforce-eager \
    --trust-remote-code \
    --enable-prefix-caching false  # CRITICAL: incompatible with KV compression

# Standard OpenAI API call—no client changes needed
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "<model_path>",
        "messages": [{"role": "user", "content": "Solve: ..."}]
    }'

Critical configuration explained: The --enable-prefix-caching false flag is non-negotiable. Prefix caching assumes KV entries are stable and reusable, but TriAttention's compressed representations change dynamically. Enabling it causes incorrect cache hits on stale compressed entries, producing gibberish outputs. The --enforce-eager flag disables CUDA graph capture, necessary because compression triggers introduce dynamic control flow that graphs cannot capture.

Example 2: Python API for Programmatic Control

For applications needing direct integration without HTTP overhead:

from triattention.vllm.runtime.integration_monkeypatch import (
    install_vllm_integration_monkeypatches,
)

# MUST call before creating LLM—patches vLLM's internal scheduler and worker
install_vllm_integration_monkeypatches(patch_scheduler=True, patch_worker=True)

# Standard vLLM API—compression happens transparently downstream
from vllm import LLM, SamplingParams

llm = LLM(
    model="<model_path>",
    dtype="bfloat16",
    max_model_len=32768,      # Maximum sequence length to support
    enforce_eager=True,       # Required: CUDA graphs incompatible with compression
    trust_remote_code=True,   # Needed for custom model architectures
)

# Generate with normal sampling parameters
outputs = llm.generate(
    ["Your prompt here"],
    SamplingParams(temperature=0.6, top_p=0.95)
)
print(outputs[0].outputs[0].text)

Why monkeypatching? TriAttention integrates at vLLM's scheduler and worker level—below the public API surface. The install_vllm_integration_monkeypatches function dynamically replaces key methods in vLLM's execution engine to inject compression logic at the exact points where KV cache allocation and attention computation occur. This preserves full API compatibility while enabling zero-overhead compression.

Example 3: Chat-Optimized Server Configuration

Interactive workloads need different tuning than batch benchmarks:

# Path to precomputed frequency statistics—required for scoring keys
export TRIATTN_RUNTIME_SPARSE_STATS_PATH=triattention/vllm/stats/qwen3_32b_int4_stats.pt

# Larger budget for multi-turn chat: 12k vs default 2048
# Conversations accumulate context; aggressive eviction destroys coherence
export TRIATTN_RUNTIME_KV_BUDGET=12000

vllm serve <model_path> \
    --dtype bfloat16 \
    --max-model-len 32768 \
    --enforce-eager \
    --trust-remote-code \
    --enable-prefix-caching false \
    --max-num-batched-tokens 1024  # Limit prefill chunk size

The 1024-token limit is crucial for chat: Large prefill chunks can inject thousands of tokens before compression triggers, temporarily exceeding the KV budget and causing OOM. By capping chunk size, you ensure compression runs frequently enough to maintain the budget invariant. For the AIME25 benchmark, this wasn't necessary—batch processing has predictable patterns. Conversations are chaotic; this constraint prevents chaos from killing your server.

Example 4: DGX Spark Verification Commands

After starting your server, verify TriAttention activation:

# Check vLLM logs for activation message:
# [TriAttention] Runtime (V2) plugin activated: patch_scheduler=True patch_worker=True

# Verify API responsiveness
curl http://127.0.0.1:8000/v1/models

# Test generation with minimal prompt
curl http://127.0.0.1:8000/v1/completions \
    -H 'Content-Type: application/json' \
    -d '{
        "model": "Qwen/Qwen3-8B",
        "prompt": "hello",
        "max_tokens": 16
    }'

Advanced Usage & Best Practices

Calibrate Custom Models Precomputed statistics exist for supported models in triattention/vllm/stats/, but custom deployments need calibration. The Calibration Guide generates Q/K frequency statistics for your specific model—critical for architectures not in the verified list.

Tune Compression Aggressiveness The TRIATTN_RUNTIME_KV_BUDGET is your primary lever. Lower values = more compression = higher speedup, but increased risk of accuracy degradation. For reasoning benchmarks, 2048-4096 hits the sweet spot. For chat, 8000-12000 preserves multi-turn coherence. Always benchmark your specific task—don't blindly copy settings.

Protect Critical Tokens Set TRIATTN_RUNTIME_PROTECT_PREFILL=true to prevent eviction of initial prompt tokens. Essential for few-shot prompts where example formatting must survive compression.

Monitor Experimental Features The TRIATTN_RUNTIME_ENABLE_EXPERIMENTAL_KV_COMPACTION and BLOCK_RECLAIM flags enable in-place compaction and freed block reclamation. Currently default-true, but monitor for stability issues in production—these touch vLLM's memory management internals.

Compose with Quantization The community llama.cpp port demonstrates 6.8x KV reduction when combined with TurboQuant. TriAttention compresses the cache structure; quantization compresses individual entries. Together they're multiplicative—this is how you run 70B-class models on 16GB cards.

Comparison with Alternatives

Why TriAttention wins: SnapKV and R-KV both rely on selecting "important" tokens via attention scores—an approximation that fails catastrophically in long reasoning chains where attention patterns are diffuse and evolving. TriAttention's trigonometric scoring bypasses selection entirely, leveraging the mathematical structure that reasoning models actually exhibit.

The throughput advantage compounds: not only is each forward pass faster due to smaller KV cache, but the memory savings enable larger batch sizes. The 6.3x MATH-500 speedup versus 2.5x AIME25 reflects this—shorter sequences allow more aggressive batching when memory isn't the bottleneck.

Frequently Asked Questions

Does TriAttention work with my GPU? Verified on NVIDIA Ampere and newer (RTX 30-series, A100, H100). Community ports support AMD via ROCm and Apple Silicon via MLX. DGX Spark/GB10 specifically enabled.

Will it break my model's accuracy? On verified models with provided statistics, AIME25 accuracy matches full attention exactly (40.8% vs 40.8%). Always benchmark your specific task—compression interacts unpredictably with certain reasoning patterns.

How do I generate statistics for unsupported models? Follow the Calibration Guide. Requires running your model on sample data to extract Q/K frequency distributions—typically 30 minutes to 2 hours depending on dataset size.

Can I use TriAttention with SGLang instead of vLLM? Yes! SGLang backend support added April 2026. See SGLang Integration docs for configuration.

Is prefix caching ever coming? Currently incompatible due to dynamic compression invalidating cached entries. The team is exploring compression-aware caching strategies—watch the repository for updates.

What's the catch with experimental features? KV compaction and block reclamation touch vLLM internals. Stable in benchmarks, but stress-test before production deployment. Disable via environment variables if issues arise.

Can I contribute a port to my favorite framework? Absolutely—community implementations are welcomed. See existing ggml and MLX ports for patterns. Maintain your own repository; the team will link it in the community table.

Conclusion: The Future of Efficient Inference Is Trigonometric

TriAttention isn't incremental improvement—it's a paradigm shift. While competitors graft compression onto attention's existing structure, TriAttention reveals and exploits the inherent trigonometric geometry that long reasoning models already possess. The result is compression that doesn't approximate, doesn't select, doesn't compromise.

The numbers speak brutally: 10.7x memory reduction, 2.5x throughput, zero accuracy loss. But the real story is accessibility. OpenClaw deployment on 24GB GPUs. MLX support for MacBooks. Community ports to llama.cpp. This is technology that democratizes serious reasoning model deployment.

My assessment? TriAttention will become standard infrastructure within 18 months. The vLLM integration path is too seamless, the accuracy guarantees too strong, and the hardware pressure too intense for alternatives to survive. SnapKV and R-KV had their moment; trigonometric compression is the next evolution.

Start now: Clone https://github.com/WeianMao/triattention, run the quick benchmark, and experience what 10.7x compression feels like when it actually works. Your GPU—and your wallet—will thank you.