Fast-FoundationStereo: 10x Faster Zero-Shot Depth Without Retraining
Fast-FoundationStereo: 10x Faster Zero-Shot Depth Without Retraining
What if every robot, drone, and AR headset could perceive depth instantly—without ever seeing your environment during training?
For years, the stereo vision community has been trapped in a brutal trade-off. Foundation models like FoundationStereo generalize beautifully across unseen domains, but they crawl along at frame rates that make real-time applications impossible. Meanwhile, fast architectures demand painstaking per-domain fine-tuning, turning every new deployment into a weeks-long data collection nightmare. Engineers have been forced to choose: accuracy or speed, never both.
That false dilemma just died.
NVIDIA's Fast-FoundationStereo—accepted to CVPR 2026—achieves what researchers claimed was impossible: strong zero-shot generalization at real-time frame rates. We're talking 23.4 milliseconds per frame with TensorRT, over 10× faster than its predecessor, while matching zero-shot accuracy. No retraining. No domain-specific datasets. No compromises.
This isn't incremental progress. It's a category shift that could reshape robotics, autonomous navigation, and spatial computing overnight. And the best part? It's completely open-source.
What is Fast-FoundationStereo?
Fast-FoundationStereo is NVIDIA's official implementation of a family of efficient stereo matching architectures designed to solve the fundamental tension between zero-shot generalization and real-time inference speed. Born from the research labs of Bowen Wen, Shaurya Dewan, and Stan Birchfield, this work builds directly upon their earlier FoundationStereo while attacking its critical weakness: computational cost.
The core insight is devastatingly simple yet technically profound. Rather than accepting the conventional wisdom that foundation models must be computationally expensive, the team employed a divide-and-conquer acceleration strategy with three mutually reinforcing components:
Knowledge distillation compresses the original hybrid backbone into a single efficient student network, preserving representational capacity while slashing operations. Blockwise neural architecture search automatically discovers optimal cost filtering designs under strict latency budgets—reducing search complexity exponentially compared to naive approaches. Structured pruning eliminates redundancy in the iterative refinement module without catastrophic accuracy collapse.
But the innovation doesn't stop at architecture optimization. The researchers developed an automatic pseudo-labeling pipeline that curated 1.4 million in-the-wild stereo pairs from internet-scale data (specifically the Stereo4D dataset) to supplement synthetic training. This pipeline automatically filters corrupted samples—detecting subtitles, mosaics, and overly challenging images—while correcting erroneous predictions in tricky regions like sky areas.
The result? A model family that spans multiple speed-accuracy operating points, from 49.4ms PyTorch inference down to 14.0ms with TensorRT optimization, all maintaining zero-shot capabilities that previously required offline computation.
Key Features That Change Everything
10× Speedup With Zero-Shot Accuracy Preserved
The headline achievement: Fast-FoundationStereo runs over 10× faster than FoundationStereo while closely matching its zero-shot accuracy. This isn't a small speedup—it's the difference between batch offline processing and live robotic perception.
Three-Pronged Acceleration Architecture
| Component | Technical Approach | Impact |
|---|---|---|
| Knowledge Distillation | Hybrid backbone → single efficient student | Reduces model complexity while preserving features |
| Blockwise NAS | Latency-constrained cost filtering search | Exponential reduction in search complexity |
| Structured Pruning | Redundant refinement module elimination | Maintains iterative refinement benefits at lower cost |
1.4M Pseudo-Labeled Real-World Pairs
The automatic curation pipeline addresses a critical data bottleneck. Real-world stereo with ground-truth metric depth is nearly impossible to collect at scale. By generating and filtering pseudo-labels from internet stereo video, the team created training data with greater diversity and realism than pure synthetic approaches.
Flexible Speed-Accuracy Trade-offs
Six checkpoint configurations let you dial precision versus latency:
| Checkpoint | valid_iters | PyTorch (ms) | TensorRT (ms) | Memory (MB) |
|---|---|---|---|---|
23-36-37 |
8 | 49.4 | 23.4 | 653 |
23-36-37 |
4 | 41.1 | 18.4 | 653 |
20-26-39 |
8 | 43.6 | 19.4 | 651 |
20-26-39 |
4 | 37.5 | 16.4 | 651 |
20-30-48 |
8 | 38.4 | 16.6 | 646 |
20-30-48 |
4 | 29.3 | 14.0 | 646 |
Production-Ready Deployment Options
Full support for PyTorch, ONNX, and TensorRT with both single-engine and two-stage export paths. The single ONNX path eliminates complex engine splitting by replacing Triton GWC kernels with native ONNX operations.
Use Cases Where Fast-FoundationStereo Dominates
1. Autonomous Robot Navigation
Mobile robots can't wait for depth computation. Traditional stereo pipelines either require per-environment calibration and fine-tuning (slowing deployment) or run too slowly for obstacle avoidance. Fast-FoundationStereo enables immediate deployment in new environments—warehouses, hospitals, outdoor terrain—without retraining, at frame rates that support real-time path planning.
2. Drone Obstacle Detection and Avoidance
Weight-constrained drones need lightweight compute. The 14ms TensorRT mode runs comfortably on embedded Jetson platforms, providing depth perception for collision avoidance without the latency that causes crashes. Zero-shot generalization means drones handle unexpected environments—entering buildings, flying through varying lighting—without prior mapping.
3. AR/VR Spatial Mapping
Mixed reality headsets demand sub-20ms motion-to-photon latency. Previous foundation stereo models couldn't meet this budget. Fast-FoundationStereo's real-time performance enables live depth estimation for occlusion handling, hand tracking, and environmental meshing—all without requiring users to "scan" spaces beforehand.
4. Industrial Inspection and Metrology
Manufacturing lines switch between parts constantly. Fine-tuning depth models for each SKU is economically absurd. Zero-shot stereo matching lets inspection systems measure dimensions, detect defects, and guide robots across diverse products immediately. The pseudo-labeling pipeline's robustness to real-world noise translates directly to factory floor reliability.
5. Automotive ADAS and Autonomy
While not a full LiDAR replacement, Fast-FoundationStereo provides redundant depth sensing for camera-dominant approaches. Its speed enables fusion with other sensors at vehicle frame rates, while zero-shot capabilities handle construction zones, weather variations, and geographic diversity without fleet-wide model updates.
Step-by-Step Installation & Setup Guide
Prerequisites
- NVIDIA GPU with CUDA 12.4 support (tested on RTX 3090)
- Docker↗ Bright Coding Blog (recommended) or conda environment
- 8GB+ GPU memory for standard models
Option 1: Docker (Recommended for TensorRT)
The Docker path provides the cleanest environment, especially for TensorRT export and inference:
# Build the container with host networking
docker build --network host -t ffs -f docker/dockerfile .
# Launch interactive container with GPU access
bash docker/run_container.sh
The Dockerfile handles CUDA, cuDNN, TensorRT, and all Python↗ Bright Coding Blog dependencies automatically.
Option 2: pip/conda Installation
For development or custom environments:
# Create isolated Python 3.12 environment
conda create -n ffs python=3.12 && conda activate ffs
# Install PyTorch 2.6 with CUDA 12.4 and xformers for efficient attention
pip install torch==2.6.0 torchvision==0.21.0 xformers --index-url https://download.pytorch.org/whl/cu124
# Install remaining dependencies
pip install -r requirements.txt
Download Pre-trained Weights
Download checkpoints from Google Drive and organize as follows:
mkdir -p weights
cp -r /path/to/downloaded/23-36-37 weights/ # Example checkpoint
Verify Installation
Run the demo on included sample data:
python scripts/run_demo.py \
--model_dir weights/23-36-37/model_best_bp2_serialize.pth \
--left_file demo_data/left.png \
--right_file demo_data/right.png \
--intrinsic_file demo_data/K.txt \
--out_dir output/ \
--valid_iters 8
REAL Code Examples From the Repository
Example 1: Basic Depth Inference (PyTorch)
The fundamental inference pipeline takes rectified stereo pairs and produces disparity, depth, and optionally point clouds:
# scripts/run_demo.py - Core inference execution
python scripts/run_demo.py \
--model_dir weights/23-36-37/model_best_bp2_serialize.pth \
--left_file demo_data/left.png \
--right_file demo_data/right.png \
--intrinsic_file demo_data/K.txt \
--out_dir output/ \
--remove_invisible 0 \ # Keep non-overlapping regions (set 1 to mask)
--denoise_cloud 1 \ # Apply statistical outlier removal
--scale 1 \ # Full resolution; use 0.5 for 2x speedup
--get_pc 1 \ # Export colored point cloud (.ply)
--valid_iters 8 \ # Refinement iterations (4 = faster, 8 = better)
--max_disp 192 \ # Max disparity; increase only for <0.1m range
--zfar 100 # Clip point cloud beyond 100 meters
Critical implementation details: The --valid_iters flag controls the iterative refinement depth. Reducing from 8 to 4 cuts 8ms off TensorRT latency with modest quality trade-off. The --scale parameter enables dynamic resolution adjustment—vital for deploying across different GPU tiers. The intrinsic file format requires specific structure: line 1 contains the flattened 3×3 K matrix, line 2 specifies baseline in meters.
Example 2: Single ONNX Export for Simplified Deployment
For production environments, export to a single ONNX file that eliminates Triton kernel dependencies:
# Export full model with ONNX-compatible operations
python scripts/make_single_onnx.py \
--model_dir weights/23-36-37/model_best_bp2_serialize.pth \
--save_path output/ \
--height 480 \ # Must be divisible by 32
--width 640 \ # Must be divisible by 32
--valid_iters 8 \ # Match inference-time refinement
--max_disp 192 \ # Consistent with training configuration
--onnx_name fast_foundationstereo
# Convert to optimized TensorRT engine with FP16
# This halves memory and significantly boosts throughput
trtexec \
--onnx=output/fast_foundationstereo.onnx \
--saveEngine=output/fast_foundationstereo.engine \
--fp16 # Enable half-precision inference
Why this matters: The single ONNX path replaces the custom Triton Group-Wise Correlation (GWC) kernel with standard ONNX operations. This eliminates engine splitting complexity and makes the model portable to any ONNX Runtime or TensorRT environment without custom plugin dependencies.
Example 3: Single TensorRT Inference
Run optimized inference with automatic format detection:
# Auto-detects .engine or .onnx in model_dir
python scripts/run_demo_single_trt.py \
--model_dir output/ \ # Contains .engine + config.yaml
--left_file demo_data/left.png \
--right_file demo_data/right.png \
--intrinsic_file demo_data/K.txt \
--out_dir output_demo/ \
--get_pc 1 \
--remove_invisible 0 \
--denoise_cloud 1 \
--zfar 100
Preprocessing requirement: The single ONNX/TensorRT model expects pre-normalized float32 inputs with ImageNet statistics. The inference script handles this automatically, but custom integrations must apply:
# Manual normalization for custom pipelines
mean = [123.675, 116.28, 103.53] # RGB channel means
std = [58.395, 57.12, 57.375] # RGB channel std deviations
normalized = (pixel - mean) / std # Per-channel operation
Example 4: Two-Stage TensorRT (Original Export Path)
For maximum compatibility with the original architecture, use the split export around the Triton GWC kernel:
# Export two ONNX files with intermediate Triton kernel
python scripts/make_onnx.py \
--model_dir weights/23-36-37/model_best_bp2_serialize.pth \
--save_path output/ \
--height 448 \ # Slightly different default than single path
--width 640 \
--valid_iters 8 \
--max_disp 192
# Convert both stages to TensorRT with CUDA graphs for minimal launch overhead
trtexec --onnx=output/feature_runner.onnx \
--saveEngine=output/feature_runner.engine --fp16 --useCudaGraph
trtexec --onnx=output/post_runner.onnx \
--saveEngine=output/post_runner.engine --fp16 --useCudaGraph
# Run two-stage inference
python scripts/run_demo_tensorrt.py \
--onnx_dir output/ \
--left_file demo_data/left.png \
--right_file demo_data/right.png \
--intrinsic_file demo_data/K.txt \
--out_dir output/ \
--get_pc 1 \
--zfar 100
Performance tip: The --useCudaGraph flag captures the execution pattern for replay, eliminating CPU launch overhead. Essential for achieving the quoted 14-23ms latencies in production loops.
Advanced Usage & Best Practices
Latency Optimization Checklist
- Use TensorRT, not PyTorch, for production: 2× speedup minimum
- Reduce
--valid_itersto 4 when 16ms frame time is acceptable - Scale inputs with
--scale 0.5for 4× pixel reduction; upsample outputs with nearest-neighbor interpolation - Warm up before timing: First run includes CUDA kernel compilation
- Keep width < 1000 pixels: Architecture optimized for this regime; downscale aggressively
Data Quality for Best Results
- Rectification is mandatory: Epipolar lines must be horizontal. Most stereo cameras (Zed, RealSense D4XX) handle this automatically
- Never swap left/right: Left image must be from physical left camera (objects appear right-er)
- Prefer PNG over JPEG: Compression artifacts corrupt matching cost computation
- RGB optimal, monochrome acceptable: Tested successfully on IR stereo pairs
Memory-Constrained Deployment
For Jetson Nano or similar edge devices:
- Use
20-30-48checkpoint (lowest memory: 646MB) - Export with
--height 320 --width 480(divisible by 32) - Reduce
--max_dispto 128 unless sub-10cm sensing required
Comparison with Alternatives
| Capability | Fast-FoundationStereo | FoundationStereo | RAFT-Stereo | Selective-IGEV |
|---|---|---|---|---|
| Zero-shot generalization | ✅ Strong | ✅ Strongest | ❌ Requires fine-tuning | ❌ Requires fine-tuning |
| Real-time speed | ✅ 14-23ms | ❌ ~230ms | ⚠️ Moderate | ⚠️ Moderate |
| No per-domain training | ✅ | ✅ | ❌ | ❌ |
| TensorRT deployment | ✅ Single & two-stage | ❌ Complex | ⚠️ Community | ⚠️ Community |
| Open-source weights | ✅ | ✅ | ✅ | ✅ |
| CVPR publication | 2026 | Earlier | 2021 | 2024 |
When to choose Fast-FoundationStereo: Any production scenario requiring immediate deployment across diverse environments without retraining—robotics fleets, consumer AR, multi-site industrial inspection.
When to accept FoundationStereo: Offline batch processing where maximum accuracy trumps speed, such as photogrammetry or scientific measurement.
FAQ
Is Fast-FoundationStereo free for commercial use?
Yes, the repository is open-source. Contact Bowen Wen (bowenw@nvidia.com) for specific commercial licensing questions, particularly regarding redistribution of trained weights.
What hardware do I need for real-time performance?
An RTX 3090 or better achieves 14-23ms with TensorRT. Jetson AGX Orin should handle reduced-resolution configurations. The model requires CUDA-capable NVIDIA GPUs; no CPU-only inference path exists.
Can I fine-tune on my own data?
The repository focuses on inference. For fine-tuning, refer to the training code structure inherited from FoundationStereo. The pseudo-labeling pipeline code may be released separately.
Why is my first inference so slow?
PyTorch JIT compilation and CUDA kernel caching cause initial overhead. Run 5-10 warmup iterations before measuring latency, or use TensorRT with pre-built engines.
How does this compare to LiDAR?
Stereo is passive, lower-cost, and higher-resolution than LiDAR, but less precise at long range and struggles in textureless regions. Fast-FoundationStereo narrows the accuracy gap while maintaining stereo's cost and power advantages.
What about dynamic scenes with moving objects?
Standard stereo matching assumes static scenes during capture. For dynamic environments, consider temporal filtering or fusion with optical flow. The zero-shot generalization helps with novel object categories.
Can I use this for monocular depth estimation?
No—this requires calibrated stereo pairs. For monocular depth, explore MiDaS, ZoeDepth, or NVIDIA's own FoundationPose extensions.
Conclusion
Fast-FoundationStereo doesn't just optimize a model—it eliminates a false choice that has constrained computer vision deployment for years. The research community long accepted that zero-shot generalization required computational profligacy, that real-time performance demanded domain-specific sacrifices. NVIDIA's CVPR 2026 breakthrough proves both assumptions wrong.
The technical architecture—knowledge distillation, blockwise NAS, structured pruning, and internet-scale pseudo-labeling—creates a reproducible template for accelerating other foundation models without accuracy collapse. The 10× speedup isn't magic; it's methodical systems engineering applied to a genuinely hard problem.
For practitioners, the implications are immediate. Deploy depth perception today without collecting training data. Ship robots that adapt to new warehouses overnight. Build AR experiences that map spaces in real-time rather than requiring tedious scanning rituals.
The code is open. The weights are available. The excuses just ran out.
Star the repository, run the demo, and join the real-time perception revolution: github.com/NVlabs/Fast-FoundationStereo
Found this breakdown valuable? Share it with your robotics, CV, or graphics team—the next project you save weeks on might be your own.
Explore on the BrightCoding network
Hand-picked resources from our other sites.
nv-tlabs/kimodo: Generate 3D Human & Robot Motion from Text Prompts
nv-tlabs/kimodo is NVIDIA's open-source kinematic motion diffusion model for generating controllable 3D human and robot motion from text prompts and constraints...
Stop Wasting Hours on Fine-Tuning! DINOv3 Is the Vision Foundation Model That Just Works
Discover DINOv3, Meta FAIR's revolutionary vision foundation model that delivers state-of-the-art dense features without fine-tuning. Learn installation, real c...
visomaster/VisoMaster: Video Face Swapping with Multi-Face Masking
visomaster/VisoMaster is an open-source Python tool for video face swapping with multi-face masking, DeepFaceLab model support, and TensorRT GPU acceleration. F...
Continuez votre lecture
Why Chandra is the Ultimate OCR Tool for Handwriting and Tables
Why Building LLM Applications From Scratch is a Game Changer
How Building LLM Apps From Scratch Changes the Future of AI Development
Stop Coding Alone: OPC-Skills Gives Your AI Agent Superpowers
Commentaires 0
Aucun commentaire pour l'instant. Soyez le premier à réagir !