wangzhiyaoo/SVFR: Unified Video Face Restoration Framework

wangzhiyaoo/SVFR: Unified Video Face Restoration Framework

Video face restoration has long required developers to stitch together separate tools for different tasks—one model for blur removal, another for colorization, yet another for inpainting. wangzhiyaoo/SVFR eliminates this fragmentation by unifying blind face restoration (BFR), colorization, and inpainting within a single diffusion-based framework. Built on Stable Video Diffusion and designed for practical deployment, SVFR lets you tackle single tasks or arbitrary combinations through one consistent inference pipeline.

What is wangzhiyaoo/SVFR?

wangzhiyaoo/SVFR is an open-source Python^{↗ Bright Coding Blog} framework for generalized video face restoration. As of January 2025, the repository has accumulated 861 stars and 90 forks on GitHub, with active development continuing through January 19, 2025. The project is maintained by Zhiyao Wang and collaborators, with the corresponding research paper published on arXiv (2501.01235).

SVFR sits at the intersection of computer vision, generative AI, and video processing. Unlike task-specific solutions that require separate model weights and preprocessing pipelines, SVFR handles multiple restoration objectives through a shared architecture. The framework builds upon Sonic's architecture and leverages Stable Video Diffusion as its generative backbone, specifically the stable-video-diffusion-img2vid-xt variant.

The project's relevance stems from a genuine engineering pain point: video face restoration workflows typically involve chaining disparate models with incompatible preprocessing requirements, temporal consistency issues, and redundant compute overhead. SVFR's unified approach reduces this complexity while maintaining or improving output quality across tasks. The availability of both local inference scripts and a Hugging Face Spaces demo makes it accessible for experimentation and integration.

Key Features

Multi-task unification is SVFR's defining characteristic. The framework supports:

• Blind Face Restoration (BFR): Recovering facial details from low-quality, degraded video without requiring reference images
• Colorization: Adding realistic color to grayscale or faded facial regions in video
• Inpainting: Filling masked or corrupted facial areas with temporally coherent content
• Arbitrary combinations: Running any subset of tasks simultaneously (BFR + colorization, BFR + colorization + inpainting, etc.)

Temporal consistency is engineered into the architecture rather than treated as an afterthought. Because SVFR operates on video sequences through Stable Video Diffusion's latent video representation, it naturally maintains frame-to-frame coherence—critical for avoiding flickering artifacts common in frame-wise image restoration approaches.

Flexible input handling includes an optional --crop_face_region flag that preprocesses input by isolating facial areas. This both improves restoration quality and reduces computational load when full-frame processing is unnecessary.

Identity preservation is implemented through dedicated components: the framework uses an identity linear layer (id_linear.pth) and InsightFace features (insightface_glint360k.pth) to maintain subject recognizability across restoration operations—a common failure mode in aggressive face enhancement.

The modular checkpoint system separates concerns: face alignment (YOLOFace), restoration UNet, identity encoding, and the base video diffusion model can be updated or swapped independently.

Use Cases

Archival video digitization represents a primary application. Historical footage often suffers from combined degradation: low resolution, color fading, and physical damage (scratches, mold). SVFR's ability to address BFR, colorization, and inpainting in one pass—rather than three sequential operations with potential error accumulation—streamlines preservation workflows.

Video conferencing enhancement benefits from the BFR pathway alone. Real-time or post-processing enhancement of compressed, low-bitrate facial video can recover detail lost to aggressive codecs without requiring sender-side modifications.

Content restoration for broadcasting leverages combined task execution. A damaged interview segment might need blur removal (BFR), conversion from black-and-white (colorization), and repair of tape-dropout artifacts (inpainting). SVFR's unified inference avoids the quality degradation that occurs when separate models with different inductive biases process the same content serially.

Research and prototyping is facilitated by the Hugging Face demo and straightforward local setup. ML practitioners can evaluate the approach against proprietary or alternative methods without extensive engineering investment.

Forensic and security applications (subject to license constraints) may use BFR for enhancing low-quality surveillance footage, though the non-commercial restriction on pretrained models limits production deployment in commercial security contexts.

Installation & Setup

SVFR requires Python 3.9 and a GPU with 16GB+ VRAM. The setup process involves three phases: environment creation, dependency installation, and checkpoint acquisition.

Environment Creation

conda create -n svfr python=3.9 -y
conda activate svfr

PyTorch Installation

Install PyTorch with CUDA support appropriate to your hardware. The README specifies this version as an example:

pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2

Verify your CUDA version with nvcc --version or nvidia-smi and adjust the PyTorch build accordingly.

Dependency Installation

pip install -r requirements.txt

Checkpoint Downloads

Stable Video Diffusion base model (requires git-lfs):

conda install git-lfs
git lfs install
git clone https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt models/stable-video-diffusion-img2vid-xt

SVFR-specific weights (manual download from Google Drive):

Place files in this structure:

└── models
    ├── face_align
    │   ├── yoloface_v5m.pt
    ├── face_restoration
    │   ├── unet.pth
    │   ├── id_linear.pth
    │   ├── insightface_glint360k.pth
    └── stable-video-diffusion-img2vid-xt
        ├── vae
        ├── scheduler
        └── ...

For local Gradio demo support: pip install gradio

Real Code Examples

Basic Single-Task Inference

The following command runs blind face restoration on a single video:

python3 infer.py \
 --config config/infer.yaml \
 --task_ids 0 \
 --input_path ./assert/lq/lq1.mp4 \
 --output_dir ./results/ \
 --crop_face_region

Explanation: --task_ids 0 selects BFR. The --crop_face_region flag preprocesses the input to focus compute on facial regions. The config file (infer.yaml) specifies model paths and inference hyperparameters. Output writes to ./results/.

Multi-Task Combined Inference

This example runs BFR, colorization, and inpainting simultaneously:

python3 infer.py \
 --config config/infer.yaml \
 --task_ids 0,1,2 \
 --input_path ./assert/lq/lq3.mp4 \
 --output_dir ./results/ \
 --mask_path ./assert/mask/lq3.png \
 --crop_face_region

Explanation: --task_ids 0,1,2 activates all three tasks. The --mask_path argument provides an inpainting mask; this is required when task 2 (inpainting) is included. The mask format should match the video dimensions.

Task ID Reference

Local Gradio Demo

python3 demo.py

This launches a web interface for interactive experimentation, mirroring the functionality of the community-hosted Hugging Face demo.

Advanced Usage & Best Practices

VRAM management: The 16GB recommendation is a minimum, not a target. For longer sequences or higher resolutions, consider processing in overlapping temporal chunks and blending at boundaries. The crop_face_region flag substantially reduces memory pressure when applicable.

Mask preparation for inpainting: The README specifies PNG format masks but does not detail color conventions. Standard practice in diffusion inpainting uses white for regions to inpaint and black for preserved content; verify this against the actual inference code if results are unexpected.

Temporal chunking strategy: For videos exceeding memory constraints, process overlapping segments (e.g., 16-frame chunks with 4-frame overlap) and use optical flow or simple crossfade blending to smooth transitions. SVFR's diffusion prior provides some tolerance for boundary inconsistencies.

Identity-sensitive applications: The identity preservation components (id_linear.pth, insightface_glint360k.pth) add computational overhead. For applications where subject recognition is unimportant (e.g., anonymous crowd footage), ablation studies without these components may yield faster inference—though this requires code modification not documented in the README.

Task combination ordering: The framework handles arbitrary task combinations, but consider whether serial processing with intermediate inspection might benefit quality-critical workflows. Combined inference is more efficient; sequential processing allows human-in-the-loop quality gates.

Comparison with Alternatives

SVFR's differentiation is clear: it occupies a niche for unified, temporally-coherent facial video restoration that neither image-specific face restoration tools nor general video super-resolution methods address directly. The trade-off is increased setup complexity and VRAM requirements. For pure BFR on images, GFPGAN or CodeFormer remain simpler alternatives. For non-facial video enhancement, BasicVSR++ and derivatives are more appropriate.

FAQ

What GPU do I need? A GPU with 16GB+ VRAM is recommended; below this, expect out-of-memory errors or need for aggressive temporal chunking.

Can I use SVFR commercially? The code is MIT-licensed. However, pretrained models are restricted to non-commercial research use only.

Does it work on CPU? The README specifies GPU requirements; CPU inference is not documented and would likely be impractically slow for video.

What video formats are supported? The examples use .mp4; specific codec support depends on OpenCV/PyAV capabilities in the environment.

How do I create inpainting masks? The README accepts PNG masks via --mask_path but does not specify creation tools. Standard image editors or programmatic mask generation (e.g., segmentation masks) are compatible approaches.

Is real-time processing possible? The diffusion-based architecture and VRAM requirements suggest inference is measured in seconds-to-minutes per video, not real-time.

Who maintains the Hugging Face demo? Community contributor @fffiloni hosts the online demo; it is not officially maintained by the paper authors.

Conclusion

wangzhiyaoo/SVFR addresses a genuine gap in the video restoration toolkit: the need for unified, temporally-consistent facial enhancement across multiple degradation types. Its 861 stars and active development reflect real interest, while the Hugging Face demo lowers the barrier for initial evaluation.

The framework best serves researchers exploring multi-task video restoration, archivists processing degraded historical footage, and engineers building preprocessing pipelines where facial quality matters. The non-commercial model license is a significant constraint for production deployment—evaluate alternatives if commercial use is required.

For the latest code, checkpoint links, and community updates, visit the repository directly:

https://github.com/wangzhiyaoo/SVFR

If you're working with related diffusion-based video tools, you may also find our coverage of [INTERNAL_LINK: stable-video-diffusion-ecosystem] relevant for broader context on the underlying generative model architecture.