Netflix Just Open-Sourced VOID: The AI That Erases Objects From Videos

Netflix Just Open-Sourced VOID: The AI That Erases Objects From Videos

What if you could remove a person from a video—and watch the objects they were holding fall naturally to the ground, as if that person never existed? Not a crude cut-and-paste job. Not a static Photoshop freeze-frame. Real, physically consistent video inpainting where cause and effect ripple through every frame.

This isn't science fiction. Netflix just dropped VOID into the open-source wild, and it's about to change how developers think about video editing forever.

For years, video object removal meant one of two painful paths: expensive manual rotoscoping by VFX artists, or brittle AI tools that left ghostly artifacts, smeared backgrounds, and zero understanding of physics. Remove a person holding a guitar, and traditional tools would either freeze the guitar in mid-air or blur it into oblivion. VOID solves this with a radically different approach—it's interaction-aware, meaning it understands and simulates the physical consequences of removal.

Built on top of the powerful CogVideoX architecture and fine-tuned specifically for video inpainting with what the researchers call "interaction-aware mask conditioning," VOID doesn't just delete pixels. It reasons about what should happen next. The result? Videos where removed objects leave behind physically plausible consequences—falling items, settling dust, natural motion that fools the eye.

In this deep dive, I'll walk you through everything you need to know: what makes VOID tick, how to set it up on your own hardware, real code examples straight from Netflix's repository, and pro tips for getting cinema-quality results. Whether you're building the next generation of video editing tools, researching computer vision, or simply obsessed with cutting-edge AI, this guide has you covered.

What is VOID?

VOID (Video Object and Interaction Deletion) is an open-source deep learning model developed by researchers at Netflix in collaboration with INSAIT at Sofia University. The project represents a significant leap forward in video inpainting technology, specifically addressing a long-neglected problem: modeling physical interactions when objects are removed from dynamic scenes.

The research team—Saman Motamed, William Harvey, Benjamin Klein, Luc Van Gool, Zhuoning Yuan, and Ta-Ying Cheng—published their work with an accompanying arXiv paper and immediately made the full implementation available on GitHub. This isn't a research teaser or a partial release. It's production-grade code with training pipelines, inference scripts, and even data generation tools for creating your own training datasets.

VOID is built on CogVideoX-Fun-V1.5-5b-InP, a capable video generation model from Alibaba, which it fine-tunes with a novel two-pass architecture. The first pass handles base inpainting with quadmask conditioning; the second pass refines temporal consistency using optical flow-warped noise. This dual-pass design is crucial for maintaining coherent motion across longer video sequences without the flickering and inconsistency that plague single-pass approaches.

What makes VOID genuinely different from predecessors is its quadmask system. Instead of a simple binary mask (object vs. background), VOID uses a four-value semantic mask that encodes: the primary object to remove, regions affected by physical interactions, overlap zones, and protected background. This richer conditioning signal allows the model to learn—and generate—physically plausible consequences of removal.

The timing couldn't be better. With the explosion of AI-generated video content, tools that can intelligently edit existing footage are becoming essential infrastructure. Netflix's decision to open-source VOID signals both technical confidence and strategic investment in the broader AI video ecosystem.

Key Features That Make VOID Insanely Powerful

Let's break down the technical innovations that separate VOID from every video inpainting tool that came before it.

Interaction-Aware Inpainting. This is VOID's killer feature. Traditional inpainting models treat removal as a hole-filling problem: given missing pixels, generate plausible content. VOID elevates this to physical reasoning: given a removed object, generate the consequences. When you delete a person leaning against a table, the table doesn't just stay put—it might shift, settle, or cause a chain reaction to nearby objects. VOID learns these dynamics from training data and reproduces them at inference time.

Two-Pass Architecture for Temporal Coherence. Pass 1 generates the base inpainting using the quadmask-conditioned transformer. Pass 2 then takes this output, warps its latent representation using optical flow, and runs a refinement pass with a separate model trained specifically for temporal consistency. This is especially critical for longer sequences where small errors compound into jarring flicker.

Quadmask Semantic Segmentation. The four-value mask format (0, 63, 127, 255) provides nuanced control:

• 0 = primary removal target
• 63 = overlap regions requiring special handling
• 127 = physically affected regions (the secret sauce)
• 255 = protected background

This design emerged from careful analysis of failure modes in binary-mask approaches and enables precise control over what the model should reconstruct versus simulate.

Automated Mask Generation Pipeline. VOID ships with a complete VLM-MASK-REASONER system that combines SAM2/3 segmentation with Google's Gemini for reasoning about interactions. Click on objects in a GUI, and the pipeline automatically identifies not just the object but everything it physically affects.

Flexible Training Infrastructure. The full training code is included, with data generation pipelines for both human-object interactions (via HUMOTO + Blender) and object-only physics (via Kubric). This means you can fine-tune VOID for your specific domain—sports footage, industrial video, medical imaging—without starting from scratch.

Production-Grade Memory Optimization. With support for model CPU offloading, sequential CPU offloading, and quantized float8 inference, VOID squeezes onto hardware that would choke on naive implementations. The default model_cpu_offload_and_qfloat8 mode makes 40GB VRAM feasible for high-resolution outputs.

Use Cases Where VOID Absolutely Dominates

Film and Television Post-Production. The obvious starting point. Remove stunt rigging, crew members in reflections, unwanted product placements, or even entire characters from scenes—while maintaining physically consistent backgrounds. The interaction awareness means a removed wire doesn't leave a floating actor; gravity takes over naturally.

Sports Broadcasting. Erase referees, camera operators, or on-field equipment from replays. More impressively, remove players to analyze team formations without visual obstruction, or create "ghost" training footage showing ideal positioning.

Autonomous Vehicle Training Data. Clean up training datasets by removing inconsistent elements (other vehicles, pedestrians) while preserving road surface reactions, debris movement, and environmental physics. This generates more controlled synthetic scenarios for edge-case training.

E-commerce and Product Visualization. Show products in isolation without the hand that placed them, the stand that held them, or the tools that assembled them—while maintaining natural shadows, surface compression, and environmental interactions that sell realism.

Scientific and Industrial Video Analysis. Remove instrumentation from experimental footage while preserving the physical phenomena being studied. A wind tunnel test with support structures removed, but airflow patterns intact. A materials test with grips erased, but stress propagation visible.

Privacy-Preserving Video Redaction. Go beyond blurring faces. Completely remove individuals from security footage, public recordings, or documentary material—with their physical effects on the environment naturally resolved, eliminating the telltale artifacts that draw attention to redaction.

Step-by-Step Installation & Setup Guide

Ready to run VOID yourself? Here's the complete setup process, extracted directly from the official repository.

Prerequisites

You'll need a GPU with 40GB+ VRAM—an NVIDIA A100 is the documented target. The model can run with memory optimizations enabled, but this is genuinely demanding inference.

Step 1: Clone and Install Dependencies

# Clone the VOID repository
git clone https://github.com/Netflix/void-model.git
cd void-model

# Install Python↗ Bright Coding Blog dependencies
pip install -r requirements.txt

Step 2: Configure Gemini API for Mask Generation

Stage 1 of the mask pipeline uses Google's Gemini for visual reasoning. Set your key:

export GEMINI_API_KEY=your_key_here

Step 3: Install SAM2 and SAM3 for Segmentation

# SAM2 (Segment Anything 2)
git clone https://github.com/facebookresearch/sam2.git
cd sam2 && pip install -e .

# SAM3 (latest iteration)
git clone https://github.com/facebookresearch/sam3.git
cd sam3 && pip install -e .

Step 4: Download the Base Inpainting Model

# Using HuggingFace CLI (install with: pip install huggingface-cli)
hf download alibaba-pai/CogVideoX-Fun-V1.5-5b-InP \
    --local-dir ./CogVideoX-Fun-V1.5-5b-InP

This downloads the pretrained CogVideoX model that VOID fine-tunes from. The inference scripts expect it at ./CogVideoX-Fun-V1.5-5b-InP by default.

Step 5: Download VOID Checkpoints

Grab both transformer checkpoints from HuggingFace:

Place these in your repository root.

Step 6: Fix ffmpeg Path (if needed)

# Create symlink to bundled ffmpeg if system ffmpeg unavailable
ln -sf $(python -c "import imageio_ffmpeg; print(imageio_ffmpeg.get_ffmpeg_exe())") ~/.local/bin/ffmpeg

Quick Verification

The fastest validation is Netflix's included Colab notebook—one click handles setup, downloads, and runs sample inference:

REAL Code Examples From the Repository

Let's examine actual code patterns from VOID's implementation, with detailed explanations of what's happening under the hood.

Example 1: Pass 1 Base Inference

This is your bread-and-butter inference command. It runs the primary inpainting model with quadmask conditioning:

python inference/cogvideox_fun/predict_v2v.py \
    --config config/quadmask_cogvideox.py \
    --config.data.data_rootdir="path/to/data_rootdir" \
    --config.experiment.run_seqs="my-video" \
    --config.experiment.save_path="path/to/output" \
    --config.video_model.model_name="path/to/CogVideoX-Fun-V1.5-5b-InP" \
    --config.video_model.transformer_path="path/to/void_pass1.safetensors"

What's happening here? The predict_v2v.py script loads the CogVideoX-Fun architecture, swaps in the fine-tuned VOID Pass 1 weights, and processes your video sequence. The quadmask_cogvideox.py config file defines critical hyperparameters: the 384×672 output resolution, 197-frame maximum sequence length, and the 85-frame temporal window for multidiffusion processing.

The run_seqs parameter accepts comma-separated values for batch processing: "video1,video2,video3". The model encodes your input video through the VAE, conditions the diffusion process with your quadmask, and denoises over 50 steps with classifier-free guidance scale of 1.0.

Pro tip: The model_cpu_offload_and_qfloat8 memory mode (configurable via --config.system.gpu_memory_mode) is your friend on tight VRAM. It trades ~15% speed for fitting on smaller GPUs.

Example 2: Pass 2 Warped Noise Refinement

For longer sequences or when you notice temporal flicker, chain into Pass 2:

python inference/cogvideox_fun/inference_with_pass1_warped_noise.py \
    --video_name my-video \
    --data_rootdir path/to/data_rootdir \
    --pass1_dir path/to/pass1_outputs \
    --output_dir path/to/pass2_outputs \
    --model_checkpoint path/to/void_pass2.safetensors \
    --model_name path/to/CogVideoX-Fun-V1.5-5b-InP

The magic here: Pass 2 computes optical flow between frames of your Pass 1 output, warps the latent representations accordingly, and uses these warped latents as initialization noise for a second diffusion process. This breaks the independent-frame assumption that causes flicker in single-pass methods.

Notice the higher default guidance scale (6.0 vs. 1.0 in Pass 1)—the warped initialization provides stronger structural prior, allowing more aggressive guidance without artifacts. The --use_quadmask True default maintains consistent conditioning across both passes.

For batch processing, edit inference/pass_2_refine.sh with your paths and run:

bash inference/pass_2_refine.sh

Example 3: Mask Generation Pipeline

Before inference, you need that crucial quadmask. Here's the automated pipeline:

# Step 0: Interactive point selection
python VLM-MASK-REASONER/point_selector_gui.py

# Steps 1-4: Full automated pipeline
bash VLM-MASK-REASONER/run_pipeline.sh my_config_points.json

Your JSON config specifies what to remove:

{
  "videos": [
    {
      "video_path": "path/to/video.mp4",
      "output_dir": "path/to/output/folder",
      "instruction": "remove the person"
    }
  ]
}

The four-stage pipeline: SAM2 generates initial segmentation masks from your clicked points. Gemini VLM analyzes the scene to identify physically affected regions. Grey masks encode interaction zones. Finally, everything combines into the quadmask format.

Optional flags for the pipeline:

bash VLM-MASK-REASONER/run_pipeline.sh my_config_points.json \
    --sam2-checkpoint path/to/sam2_hiera_large.pt \
    --device cuda

Example 4: Manual Mask Refinement

When automation isn't perfect, the GUI editor gives pixel-precise control:

python VLM-MASK-REASONER/edit_quadmask.py

Load a sequence with input_video.mp4 and quadmask_0.mp4 side-by-side. The grid toggles let you rapidly mark interaction regions, while brush tools handle fine detail. The "Copy from Previous Frame" feature accelerates propagation through sequences with consistent object positions.

Critical workflow: Navigate with arrow keys, undo/redo with Ctrl+Z/Y, save to overwrite the quadmask, then re-run Pass 1 inference. Iterate until physical plausibility is perfect.

Example 5: Training Your Own VOID

For researchers and domain specialists, here's Pass 1 training launch:

bash scripts/cogvideox_fun/train_void.sh

Key training arguments include:

• --train_mode="void" — enables the specialized inpainting objective
• --use_quadmask — activates four-value mask conditioning
• --use_vae_mask — encodes masks through the VAE for latent-space consistency
• --learning_rate 1e-5 — conservative fine-tuning from CogVideoX base

Pass 2 training requires your completed Pass 1 checkpoint:

TRANSFORMER_PATH=path/to/pass1_checkpoint.safetensors \
    bash scripts/cogvideox_fun/train_void_warped_noise.sh

The --use_warped_noise flag enables optical flow warping, with --warped_noise_degradation 0.3 controlling how much pure noise blends with warped initialization—higher values preserve more original motion, lower values allow more correction.

Advanced Usage & Best Practices

Resolution vs. Length Trade-offs. VOID's 384×672 default balances quality and memory. For 40GB VRAM, this at 197 frames is near the limit. Need longer sequences? Drop to 256×448 or use the temporal windowing (85 frames processed at a time with overlap). Need higher resolution? Shorten sequences or use sequential_cpu_offload at significant speed cost.

Prompt Engineering for Backgrounds. The prompt.json "bg" field describes the desired final scene, not the removal operation. Be specific about what remains: "A wooden table with scattered coffee beans" outperforms "A table". The model uses this for classifier-free guidance conditioning—richer prompts reduce ambiguity in generation.

Pass 2 Thresholds. Not every video needs Pass 2. Static camera, short clips, and simple backgrounds often look perfect after Pass 1. Save time by evaluating Pass 1 first; apply Pass 2 selectively for complex motion, long sequences, or when you spot temporal inconsistency.

Custom Training Data Generation. The HUMOTO pipeline requires Blender setup but produces unmatched human-object interaction diversity. For object-only physics, Kubric runs immediately with pip install kubric. Start with Kubric for proof-of-concept, invest in HUMOTO for production quality.

Memory Optimization Hierarchy: model_full_load (fastest, needs 80GB+) → model_cpu_offload (moderate speed, 60GB) → model_cpu_offload_and_qfloat8 (slowest, 40GB). Profile your hardware and choose accordingly.

Comparison with Alternatives

VOID occupies a unique position: fully open-source with training infrastructure, yet achieving interaction complexity that previously required expensive manual work or proprietary film pipelines. The trade-off is computational intensity—this is not real-time, and not lightweight.

FAQ

What hardware do I absolutely need for VOID?

A GPU with 40GB+ VRAM. NVIDIA A100 is documented and tested. With aggressive memory optimization (sequential_cpu_offload), you might squeeze onto 32GB, but expect significant slowdown. Consumer GPUs (24GB RTX 4090) are not officially supported.

Can VOID run on CPU or Apple Silicon?

Not practically. The diffusion model requires CUDA-optimized operations, and inference times would stretch to impractical durations. Stick to NVIDIA datacenter GPUs or cloud instances.

How does VOID differ from Photoshop's Generative Fill for video?

Adobe's tools process frames independently or with limited temporal awareness. VOID's interaction-aware quadmask and two-pass flow warping explicitly model physical consequences across time. Remove a person holding a guitar in Photoshop, and the guitar likely disappears or freezes. VOID makes it fall.

Is the training data available?

Due to licensing constraints on HUMOTO, Netflix releases the data generation code rather than pre-built datasets. You must request HUMOTO access from Adobe Research, then run the Blender pipeline. Kubric data generates immediately without external dependencies.

Can I fine-tune VOID for my specific domain?

Absolutely. The full training code is included with DeepSpeed multi-GPU support. Generate domain-specific training data (medical instruments, sports equipment, industrial machinery) using the provided pipelines, then fine-tune from the released checkpoints.

What's the output resolution limit?

384×672 is tested and recommended. The architecture supports variable resolutions, but memory scales quadratically. For 4K output, you'd need model parallelism across multiple GPUs or significant architectural modifications.

How do I handle failure cases where physics looks wrong?

Use the manual mask editor to refine quadmask regions, especially the 127 (affected region) values. Incorrect physics usually stems from under-specified interaction zones. Iterate on mask accuracy before adjusting generation parameters.

Conclusion: Why VOID Matters Now

VOID isn't just another video inpainting model. It's a statement about where AI video editing is heading—from pixel-pushing toward physical reasoning. Netflix's decision to fully open-source this, including training infrastructure and data generation tools, accelerates the entire field.

The interaction-aware paradigm will become standard. Within months, expect to see this approach integrated into commercial tools, fine-tuned for specific industries, and extended to real-time applications as hardware improves.

For developers and researchers, VOID offers immediate practical value: a working system for complex video editing tasks that previously required expensive VFX pipelines. For the broader ecosystem, it establishes open standards for how we think about object removal—not as deletion, but as physical consequence simulation.

The repository is actively maintained with community demos already emerging. The HuggingFace Gradio demo lets you experiment without local setup. The Colab notebook provides one-click access to the full pipeline.

Don't just read about the future of video editing. Go build it.

👉 Explore VOID on GitHub

👉 Try the Interactive Demo

👉 Read the Research Paper