HunyuanVideo: 13B Video DiT, Flow Matching, and 3D VAE Latent Compression

Figure 1: High-fidelity video synthesis with HunyuanVideo. The 13B-parameter flow-matching Diffusion Transformer (DiT) couples full 3D spatiotemporal attention with causal 3D VAE latent compression to generate temporally consistent 1080p video streams.

The Open-Source Frontier in Video Generation

While proprietary generative video models (such as Sora, Veo 2, and Kling) establish strong quality benchmarks, their closed APIs prevent custom weight fine-tuning, domain-specific distillation, and self-hosted inference for robotics simulation.

HunyuanVideo, developed by Tencent, provides a fully open-source 13B-parameter text-to-video (T2V) and image-to-video (I2V) foundation model. The system unifies causal 3D VAE compression, multimodal LLM (MLLM) text conditioning, and rectified flow matching over full 3D spatiotemporal self-attention.

Architectural Comparison

Mathematical Formulation

Figure 2: End-to-end HunyuanVideo generation pipeline. Prompt features from HunyuanMLLM condition a 13B parameter DiT operating over causal 3D VAE latents via rectified flow trajectory integration.

1. Causal 3D VAE Latent Compression

Direct pixel-space video diffusion is computationally intractable. A 5-second $1080\times 1920$ sequence at 24 FPS contains $248.8\times 10^6$ raw values.

HunyuanVideo employs a Causal 3D VAE with a compression factor of $8\times 8\times 4$ (height $\times$ width $\times$ time), mapping input $X\in {\mathbb{R}}^{3\times T\times H\times W}$ into latent space:

$\mathbf{z}={\mathcal{E}}_{\text{3D}}(X)\in {\mathbb{R}}^{16\times \frac{T}{4}\times \frac{H}{8}\times \frac{W}{8}}$

For $T=129,H=1080,W=1920$, the latent dimensions shrink to $16\times 33\times 135\times 240$ ($\sim 1.6\times 10^6$ tokens, a $155\times$ compression). The temporal causality constraint ${\mathbf{z}}_t=\mathcal{E}(X_{\le t})$ guarantees that future frames do not leak into historical latent representations, enabling continuous temporal sliding-window extrapolation.

2. 13B DiT with 3D Rotary Position Embeddings (RoPE-3D)

The transformer backbone consists of 27 blocks (hidden dimension $d=1280$, 20 attention heads, intermediate MLP dimension 3072). Full 3D attention computes dense affinities across the flattened token sequence $S=\frac{T}{4}\times \frac{H}{8}\times \frac{W}{8}$.

Positional structure is preserved via decoupled 3D Rotary Position Embeddings:

${\mathbf{R}}_{3\text{D}}(t,y,x)={\mathbf{R}}_t({\theta }_t)\oplus {\mathbf{R}}_y({\theta }_y)\oplus {\mathbf{R}}_x({\theta }_x)$

This separation enables variable-aspect-ratio rendering and duration extrapolation without re-training positional indices.

3. Rectified Flow Matching Objective

Instead of discrete Gaussian noise addition (DDPM), training minimizes the velocity field prediction error along linear optimal-transport paths:

${\mathcal{L}}_{\text{DiT}}={\mathbb{E}}_{t,{\mathbf{z}}_0,{\mathbf{z}}_1}\left[{\Vert v_{\theta }({\mathbf{z}}_t,t,c)-({\mathbf{z}}_1-{\mathbf{z}}_0)\Vert }_2^2\right]$

where ${\mathbf{z}}_t=(1-t){\mathbf{z}}_0+t{\mathbf{z}}_1$ for $t\in [0,1]$, and $c$ denotes the text embedding from HunyuanMLLM.

Implementation: PyTorch Generation & Optimization Pipeline

Environment Setup

conda create -n hyvid python=3.11 -y
conda activate hyvid

# PyTorch with CUDA 12.4
pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124

# Core dependencies
pip install transformers==4.46.0 accelerate==1.0.0 diffusers==0.31.0
pip install einops timm xformers decord imageio[ffmpeg]
pip install git+https://github.com/Tencent-Hunyuan/HunyuanVideo.git

Step 1: Text-to-Video Pipeline with Flow Shift

from __future__ import annotations

import torch
from hyvideo import HunyuanVideoPipeline
from hyvideo.utils import save_video


def generate_t2v(
    prompt: str,
    output_path: str = "output_clip.mp4",
    turbo: bool = False,
    device: str = "cuda",
) -> None:
    repo = "Tencent-Hunyuan/HunyuanVideo-Turbo" if turbo else "Tencent-Hunyuan/HunyuanVideo"

    pipe = HunyuanVideoPipeline.from_pretrained(
        repo,
        torch_dtype=torch.bfloat16,
        variant="fp16",
    ).to(device)

    # Memory optimization hooks
    pipe.enable_model_cpu_offload()
    pipe.enable_vae_tiling()

    with torch.inference_mode():
        video_frames = pipe(
            prompt=prompt,
            num_frames=129,             # 5.3 seconds @ 24 fps
            height=1080,
            width=1920,
            num_inference_steps=8 if turbo else 50,
            guidance_scale=6.0,
            flow_shift=5.0,             # Calibrated time-step shift for 1080p resolution
            output_type="pil",
        ).frames[0]

    save_video(video_frames, output_path, fps=24)
    print(f"Generated video saved to {output_path}")


if __name__ == "__main__":
    generate_t2v(
        prompt="A 6-DoF robotic manipulator sorting colored cubes on a steel table, "
               "laboratory lighting, smooth cinematic camera dolly, 4K photorealistic"
    )

Step 2: Image-to-Video (I2V) with First-Frame Conditioning

from __future__ import annotations

from PIL import Image
import torch
from hyvideo import HunyuanVideoPipeline
from hyvideo.utils import save_video


def generate_i2v(
    image_path: str,
    prompt: str,
    output_path: str = "output_i2v.mp4",
    device: str = "cuda",
) -> None:
    pipe = HunyuanVideoPipeline.from_pretrained(
        "Tencent-Hunyuan/HunyuanVideo-I2V",
        torch_dtype=torch.bfloat16,
    ).to(device)

    pipe.enable_model_cpu_offload()
    pipe.enable_vae_tiling()

    init_image = Image.open(image_path).convert("RGB").resize((1920, 1080), Image.LANCZOS)

    with torch.inference_mode():
        video_frames = pipe(
            image=init_image,
            prompt=prompt,
            num_frames=129,
            height=1080,
            width=1920,
            num_inference_steps=50,
            guidance_scale=6.5,
            output_type="pil",
        ).frames[0]

    save_video(video_frames, output_path, fps=24)


if __name__ == "__main__":
    generate_i2v(
        image_path="assets/start_pose.png",
        prompt="The robot arm smoothly reaches down, closes its gripper around the target cylinder, "
               "and lifts it vertically, cinematic slow-motion"
    )

Step 3: Autoregressive Temporal Extension (5s → 30s)

from __future__ import annotations

import torch
from hyvideo import HunyuanVideoPipeline
from hyvideo.utils import save_video


def extend_video_sequence(
    prompt: str,
    total_seconds: int = 20,
    output_path: str = "extended_sequence.mp4",
) -> None:
    pipe = HunyuanVideoPipeline.from_pretrained(
        "Tencent-Hunyuan/HunyuanVideo",
        torch_dtype=torch.bfloat16,
    ).to("cuda")
    pipe.enable_model_cpu_offload()
    pipe.enable_vae_tiling()

    fps = 24
    context_frames = 24  # 1.0 second overlapping boundary context

    # 1. Generate initial seed segment (5s)
    with torch.inference_mode():
        seed_frames = pipe(
            prompt=prompt,
            num_frames=120,
            height=1080,
            width=1920,
            num_inference_steps=40,
        ).frames[0]

    accumulated_frames = list(seed_frames)
    chunks_needed = (total_seconds - 5) // 2

    # 2. Autoregressive temporal sliding-window generation
    for chunk_idx in range(chunks_needed):
        with torch.inference_mode():
            extension = pipe(
                prompt=prompt,
                num_frames=48,  # 2-second chunk
                height=1080,
                width=1920,
                init_frames=accumulated_frames[-context_frames:],
                init_strength=0.65,  # Denoising constraint factor
                num_inference_steps=35,
            ).frames[0]

        accumulated_frames.extend(extension[context_frames:])
        print(f"Rendered {len(accumulated_frames) / fps:.1f} seconds...")

    save_video(accumulated_frames, output_path, fps=fps)


if __name__ == "__main__":
    extend_video_sequence(
        prompt="Autonomous mobile robot navigating down an industrial warehouse corridor, "
               "continuous forward tracking shot, ambient overhead fluorescent lighting",
        total_seconds=15,
    )

Empirical Benchmark Evaluation

We benchmarked HunyuanVideo against closed-source foundation models and open-source alternatives on the VBench and MovieGenBench evaluation suites:

Production Deployment on 24 GB Consumer GPUs

Executing full 3D attention over $1.6\times 10^6$ tokens exceeds 90 GB of unconstrained FP32 activations. To run inference on a single 24 GB GPU (e.g., RTX 4090 or A5000):

def configure_low_vram_stack(pipeline: HunyuanVideoPipeline) -> HunyuanVideoPipeline:
    # 1. Offload non-active layers to host RAM
    pipeline.enable_model_cpu_offload()

    # 2. Tile spatial VAE decode passes into 256x256 blocks
    pipeline.enable_vae_tiling()

    # 3. Slice multi-head attention computations
    pipeline.enable_attention_slicing(slice_size=1)

    # 4. Use memory-efficient FlashAttention / xFormers
    pipeline.enable_xformers_memory_efficient_attention()

    return pipeline

Troubleshooting Common Generation Issues

1. High-Frequency Temporal Flicker

• Symptom: Subtle luminance or texture shimmering across static backgrounds.
• Remedy: Specify concrete lighting conditions (e.g., diffuse studio illumination, static camera mount) in the prompt and verify flow_shift >= 4.0.

2. High Velocity Motion Blur

• Symptom: Fast-moving objects dissolve into blurry streaks.
• Remedy: Reduce the flow_shift hyperparameter from 5.0 to 2.5 and increase inference steps to 50.

3. Discontinuous Chunk Seams During Long Video Extension

• Symptom: Sudden camera jump or subject identity shift across autoregressive chunk boundaries.
• Remedy: Increase init_strength to 0.70 and expand the overlapping boundary context window to 32 frames.

References

1. Tencent Hunyuan Team. (2024–2026). HunyuanVideo: A Systematic Framework For Large Video Generation Model. GitHub.
2. Peebles, W., & Xie, S. (2023). Scalable Diffusion Models with Transformers. ICCV.
3. Lipman, Y., et al. (2023). Flow Matching for Generative Modeling. ICLR.
4. Zheng, Z., et al. (2024). VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR.
5. Su, J., et al. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing.
6. Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes. ICLR.