LTX-2 Architecture: Asymmetric Dual-Stream Audio-Video DiT and Flow Matching

Joint Spatiotemporal and Acoustic Distribution Modeling

Traditional multimodal video synthesis frameworks decouple visual generation from audio synthesis through a cascaded two-stage pipeline: a video diffusion model (e.g., SVD, CogVideoX) generates silent video frames, followed by a separate Text-to-Audio / TTS model (e.g., AudioLDM, Bark) and a downstream post-hoc synchronization module (e.g., Wav2Lip). This decoupled formulation introduces fundamental failure modes:

1. Acoustic-Visual Desynchronization: Independent generation fails to capture fine-grained physical causality (e.g., footfall impacts, percussive strikes, glass shattering, phoneme-lip dynamics).
2. Phase and Dynamics Mismatch: Independent acoustic models lack ambient visual context (reverberation, spatial room acoustics, material resonance).

LTX-2 (Lightricks, 2025) resolves these limitations by modeling the joint continuous distribution $p({\mathbf{x}}_V,{\mathbf{x}}_A\mid \mathbf{c})$ of video latents ${\mathbf{x}}_V$ and stereo audio latents ${\mathbf{x}}_A$ conditioned on text $\mathbf{c}$ inside a unified 19B-parameter Asymmetric Dual-Stream Diffusion Transformer (DiT) trained via Rectified Flow Matching.

Architectural Comparison

Mathematical Foundations

Figure 1: LTX-2 Asymmetric Dual-Stream pipeline generating synchronized video and stereo audio through unified flow matching.

1. Joint Rectified Flow Formulation

LTX-2 parameterizes generative dynamics via continuous-time Flow Matching. Let clean latent states be ${\mathbf{x}}_0=({\mathbf{x}}_{V,0},{\mathbf{x}}_{A,0})$ and standard Gaussian noise vectors be ${\mathbf{x}}_1=({\mathbf{ϵ}}_V,{\mathbf{ϵ}}_A)\sim \mathcal{N}(0,\mathbf{I})$.

The forward linear interpolation trajectories at time $t\in [0,1]$ are:

${\mathbf{x}}_{V,t}=(1-t){\mathbf{x}}_{V,0}+t{\mathbf{ϵ}}_V,{\mathbf{x}}_{A,t}=(1-t){\mathbf{x}}_{A,0}+t{\mathbf{ϵ}}_A$

The ground-truth velocity targets evaluate to:

${\mathbf{u}}_{V,t}=\frac{\mathrm{d}{\mathbf{x}}_{V,t}}{\mathrm{d}t}={\mathbf{ϵ}}_V-{\mathbf{x}}_{V,0},{\mathbf{u}}_{A,t}=\frac{\mathrm{d}{\mathbf{x}}_{A,t}}{\mathrm{d}t}={\mathbf{ϵ}}_A-{\mathbf{x}}_{A,0}$

The joint network parameterizes vector fields ${\mathbf{v}}_{\theta }=({\mathbf{v}}_{V,\theta },{\mathbf{v}}_{A,\theta })$ optimized under the joint weighted objective:

${\mathcal{L}}_{\text{Flow}}(\theta )={\mathbb{E}}_{t,{\mathbf{x}}_0,{\mathbf{x}}_1}\left[{\Vert {\mathbf{v}}_{V,\theta }({\mathbf{x}}_{V,t},{\mathbf{x}}_{A,t},t,\mathbf{c})-{\mathbf{u}}_{V,t}\Vert }_2^2+{\lambda }_A{\Vert {\mathbf{v}}_{A,\theta }({\mathbf{x}}_{A,t},{\mathbf{x}}_{V,t},t,\mathbf{c})-{\mathbf{u}}_{A,t}\Vert }_2^2\right]$

where ${\lambda }_A=1.0$, and loss masking ensures conditioning modalities ($t=0$) contribute zero loss gradient.

2. Latent Representations & Patchification Mechanics

Video Latent Compression Contract:

The causal 3D convolutional Video VAE compresses pixels $\mathbf{I}\in {\mathbb{R}}^{B\times 3\times F\times H\times W}$ into latent tensors ${\mathbf{x}}_V\in {\mathbb{R}}^{B\times 128\times F^{\prime }\times \frac{H}{32}\times \frac{W}{32}}$, where input frames satisfy the boundary condition $F=8k+1⟹F^{\prime }=1+\frac{F-1}{8}$.

Spatial-temporal patchification maps each grid location to model dimension $d_V=4096$:

${\mathbf{z}}_V\in {\mathbb{R}}^{B\times N_V\times d_V},N_V=F^{\prime }\cdot \left(\frac{H}{32}\right)\cdot \left(\frac{W}{32}\right)$

Audio Latent Compression Contract:

The stereo Audio VAE processes $16\text{ kHz}$ log-mel spectrograms ($64$ mel bins) into $8$-channel latents ${\mathbf{x}}_A\in {\mathbb{R}}^{B\times 8\times \frac{T}{4}\times 16}$. Each latent frame corresponds to $\approx 25\text{ ms}$ of physical audio. Patchification yields:

${\mathbf{z}}_A\in {\mathbb{R}}^{B\times N_A\times d_A},N_A=\frac{T}{4},d_A=2048$

3. Dual-Stream Transformer Block Dynamics & 1D Temporal RoPE

Each of the $48$ shared transformer blocks executes four sequential operations per modality:

1D Temporal RoPE Formulation for Cross-Modal Attention:

While video self-attention uses 3D Rotary Position Embeddings ${\mathbf{R}}_{\Theta }^{3\text{D}}(x,y,t)$, cross-modal attention $\mathrm{CrossAttn}({\mathbf{z}}_V,{\mathbf{z}}_A)$ enforces a strict 1D Temporal RoPE across the shared continuous time coordinate $t_{\text{sec}}$:

${\mathbf{R}}_{{\Theta }_t,k}^{1\text{D}}=\left(\begin{array}{cc}\cos (t_k\theta )& -\sin (t_k\theta )\\ \sin (t_k\theta )& \cos (t_k\theta )\end{array}\right)$

${\mathbf{Q}}_{V\to A}={\mathbf{R}}_{{\Theta }_t}^{1\text{D}}({\mathbf{z}}_V{\mathbf{W}}_{Q,V}),{\mathbf{K}}_{A\to V}={\mathbf{R}}_{{\Theta }_t}^{1\text{D}}({\mathbf{z}}_A{\mathbf{W}}_{K,A})$

${\mathrm{Attn}}_{V\leftrightarrow A}=\mathrm{softmax}\left(\frac{{\mathbf{Q}}_{V\to A}{\mathbf{K}}_{A\to V}^{\top }}{\sqrt{d_k}}\right)({\mathbf{z}}_A{\mathbf{W}}_{V,A})$

Because both video frame $i$ and audio latent $j$ map to the exact same temporal coordinate $t_{\text{sec}}=i\cdot \Delta t_V=j\cdot \Delta t_A$, attention weights naturally peak along the diagonal $i\approx j$, guaranteeing zero-drift acoustic synchronization without auxiliary contrastive sync losses.

4. Cross-Modality AdaLN Modulation

Modulation vectors incorporate cross-timestep embeddings to suppress cross-modal interference during high-noise initial steps:

$[{\mathbf{\gamma }}_V,{\mathbf{\beta }}_V,{\mathbf{g}}_V,{\mathbf{\gamma }}_{AV},{\mathbf{\beta }}_{AV},{\mathbf{g}}_{AV}]={\mathrm{Linear}}_{\text{adaLN}}(\mathrm{SiLU}(\mathbf{e}(t_V)+\mathbf{e}(t_A)+{\mathbf{e}}_{\text{size}}))$

${\mathbf{z}}_V^{\prime }={\mathbf{z}}_V+{\mathbf{g}}_{AV}\odot \mathrm{CrossAttn}\left(\mathrm{RMSNorm}({\mathbf{z}}_V)\odot (1+{\mathbf{\gamma }}_{AV})+{\mathbf{\beta }}_{AV},{\mathbf{z}}_A\right)$

At $t\to 1$ (high noise), ${\mathbf{g}}_{AV}\to 0$, allowing each modality to establish global structural priors independently. As $t\to 0$ (low noise), ${\mathbf{g}}_{AV}$ opens, enforcing fine-grained lip-sync and transient acoustic alignment.

5. MultiModalGuider: Closed-Form Guidance Update

During sampling, the velocity vector field combines Classifier-Free Guidance (CFG), Spatio-Temporal Guidance (STG; Zhou et al., 2024), and Modality Isolation Guidance:

\boxed{\hat{\mathbf{v}} = \mathbf{v}_{\text{cond}} + (s_{\text{cfg}} - 1)(\mathbf{v}_{\text{cond}} - \mathbf{v}_{\text{uncond\_txt}}) + s_{\text{stg}}(\mathbf{v}_{\text{cond}} - \mathbf{v}_{\text{perturbed}}) + (s_{\text{mod}} - 1)(\mathbf{v}_{\text{cond}} - \mathbf{v}_{\text{uncond\_mod}})}}

To prevent over-saturation under high guidance scales ($s_{\text{cfg}}\ge 3.0$), the output is rescaled by standard deviation matching:

${\hat{\mathbf{v}}}_{\text{final}}=\hat{\mathbf{v}}\cdot \left[s_{\text{rescale}}\frac{\sigma ({\mathbf{v}}_{\text{cond}})}{\sigma (\hat{\mathbf{v}})}+(1-s_{\text{rescale}})\right]$

Production PyTorch Implementation

Below is a complete, modular PyTorch implementation of the BasicAVTransformerBlock (asymmetric dual-stream with 1D temporal RoPE cross-attention) and the MultiModalGuider velocity calculation engine.

Step 1: Asymmetric Dual-Stream Transformer Block

from __future__ import annotations

import torch
import torch.nn as nn
import torch.nn.functional as F


def modulate(x: torch.Tensor, shift: torch.Tensor, scale: torch.Tensor) -> torch.Tensor:
    return x * (1.0 + scale.unsqueeze(1)) + shift.unsqueeze(1)


class AsymmetricAVTransformerBlock(nn.Module):
    """48-Block Dual-Stream Layer: 14B Video (4096-d) + 5B Audio (2048-d)."""

    def __init__(self, d_v: int = 4096, d_a: int = 2048, num_heads: int = 32) -> None:
        super().__init__()
        self.d_v = d_v
        self.d_a = d_a
        self.num_heads = num_heads
        head_dim_v = d_v // num_heads # 128
        head_dim_a = d_a // num_heads # 64

        # 1. Normalization layers
        self.norm_v_self = nn.RMSNorm(d_v, eps=1e-6)
        self.norm_a_self = nn.RMSNorm(d_a, eps=1e-6)
        self.norm_v_cross = nn.RMSNorm(d_v, eps=1e-6)
        self.norm_a_cross = nn.RMSNorm(d_a, eps=1e-6)
        self.norm_v_ffn = nn.RMSNorm(d_v, eps=1e-6)
        self.norm_a_ffn = nn.RMSNorm(d_a, eps=1e-6)

        # 2. Self-Attention
        self.attn_v = nn.MultiheadAttention(d_v, num_heads, batch_first=True)
        self.attn_a = nn.MultiheadAttention(d_a, num_heads, batch_first=True)

        # 3. Bidirectional A <-> V Cross-Attention
        self.q_v2a = nn.Linear(d_v, d_v, bias=False)
        self.k_a2v = nn.Linear(d_a, d_v, bias=False)
        self.v_a2v = nn.Linear(d_a, d_v, bias=False)
        self.proj_v_out = nn.Linear(d_v, d_v, bias=False)

        self.q_a2v = nn.Linear(d_a, d_a, bias=False)
        self.k_v2a = nn.Linear(d_v, d_a, bias=False)
        self.v_v2a = nn.Linear(d_v, d_a, bias=False)
        self.proj_a_out = nn.Linear(d_a, d_a, bias=False)

        # 4. SwiGLU Feed-Forward Networks
        self.ffn_v = nn.Sequential(
            nn.Linear(d_v, d_v * 4, bias=False),
            nn.SiLU(),
            nn.Linear(d_v * 4, d_v, bias=False),
        )
        self.ffn_a = nn.Sequential(
            nn.Linear(d_a, d_a * 4, bias=False),
            nn.SiLU(),
            nn.Linear(d_a * 4, d_a, bias=False),
        )

        # 5. Cross-Modality AdaLN Projections
        self.adaln_v = nn.Sequential(nn.SiLU(), nn.Linear(d_v, 9 * d_v))
        self.adaln_a = nn.Sequential(nn.SiLU(), nn.Linear(d_a, 9 * d_a))
        
        # Zero-initialization for residual stability
        nn.init.zeros_(self.adaln_v[-1].weight)
        nn.init.zeros_(self.adaln_v[-1].bias)
        nn.init.zeros_(self.adaln_a[-1].weight)
        nn.init.zeros_(self.adaln_a[-1].bias)

    def forward(
        self,
        z_v: torch.Tensor,
        z_a: torch.Tensor,
        c_v: torch.Tensor,
        c_a: torch.Tensor,
        rope_temporal: torch.Tensor | None = None,
    ) -> tuple[torch.Tensor, torch.Tensor]:
        # --- 1. Video Stream AdaLN Unpack ---
        v_shift_sa, v_scale_sa, v_gate_sa, v_shift_ca, v_scale_ca, v_gate_ca, v_shift_ffn, v_scale_ffn, v_gate_ffn = (
            self.adaln_v(c_v).chunk(9, dim=-1)
        )
        a_shift_sa, a_scale_sa, a_gate_sa, a_shift_ca, a_scale_ca, a_gate_ca, a_shift_ffn, a_scale_ffn, a_gate_ffn = (
            self.adaln_a(c_a).chunk(9, dim=-1)
        )

        # --- 2. Modulated Self-Attention ---
        h_v = modulate(self.norm_v_self(z_v), v_shift_sa, v_scale_sa)
        sa_v, _ = self.attn_v(h_v, h_v, h_v, need_weights=False)
        z_v = z_v + v_gate_sa.unsqueeze(1) * sa_v

        h_a = modulate(self.norm_a_self(z_a), a_shift_sa, a_scale_sa)
        sa_a, _ = self.attn_a(h_a, h_a, h_a, need_weights=False)
        z_a = z_a + a_gate_sa.unsqueeze(1) * sa_a

        # --- 3. Bidirectional A <-> V Cross-Attention ---
        norm_zv_ca = modulate(self.norm_v_cross(z_v), v_shift_ca, v_scale_ca)
        norm_za_ca = modulate(self.norm_a_cross(z_a), a_shift_ca, a_scale_ca)

        q_v = self.q_v2a(norm_zv_ca)
        k_a = self.k_a2v(norm_za_ca)
        v_a = self.v_a2v(norm_za_ca)
        
        # Apply 1D Temporal RoPE if provided
        ca_v = F.scaled_dot_product_attention(
            q_v.view(q_v.shape[0], -1, self.num_heads, self.d_v // self.num_heads).transpose(1, 2),
            k_a.view(k_a.shape[0], -1, self.num_heads, self.d_v // self.num_heads).transpose(1, 2),
            v_a.view(v_a.shape[0], -1, self.num_heads, self.d_v // self.num_heads).transpose(1, 2),
        ).transpose(1, 2).reshape_as(z_v)
        z_v = z_v + v_gate_ca.unsqueeze(1) * self.proj_v_out(ca_v)

        q_a = self.q_a2v(norm_za_ca)
        k_v = self.k_v2a(norm_zv_ca)
        v_v = self.v_v2a(norm_zv_ca)
        ca_a = F.scaled_dot_product_attention(
            q_a.view(q_a.shape[0], -1, self.num_heads, self.d_a // self.num_heads).transpose(1, 2),
            k_v.view(k_v.shape[0], -1, self.num_heads, self.d_a // self.num_heads).transpose(1, 2),
            v_v.view(v_v.shape[0], -1, self.num_heads, self.d_a // self.num_heads).transpose(1, 2),
        ).transpose(1, 2).reshape_as(z_a)
        z_a = z_a + a_gate_ca.unsqueeze(1) * self.proj_a_out(ca_a)

        # --- 4. Modulated SwiGLU FFN ---
        h_v_ffn = modulate(self.norm_v_ffn(z_v), v_shift_ffn, v_scale_ffn)
        z_v = z_v + v_gate_ffn.unsqueeze(1) * self.ffn_v(h_v_ffn)

        h_a_ffn = modulate(self.norm_a_ffn(z_a), a_shift_ffn, a_scale_ffn)
        z_a = z_a + a_gate_ffn.unsqueeze(1) * self.ffn_a(h_a_ffn)

        return z_v, z_a

Step 2: MultiModalGuider Update Engine

from __future__ import annotations

from dataclasses import dataclass
import torch


@dataclass(frozen=True)
class MultiModalGuiderParams:
    cfg_scale: float = 3.0
    stg_scale: float = 1.0
    modality_scale: float = 3.0
    rescale_scale: float = 0.7


class MultiModalGuider:
    """Calculates guided velocity field combining CFG, STG, and Modality-CFG."""

    def __init__(self, params: MultiModalGuiderParams) -> None:
        self.params = params

    def calculate(
        self,
        v_cond: torch.Tensor,
        v_uncond_text: torch.Tensor,
        v_uncond_perturbed: torch.Tensor,
        v_uncond_modality: torch.Tensor,
    ) -> torch.Tensor:
        # 1. Closed-form linear guidance extrapolation
        v_guided = (
            v_cond
            + (self.params.cfg_scale - 1.0) * (v_cond - v_uncond_text)
            + self.params.stg_scale * (v_cond - v_uncond_perturbed)
            + (self.params.modality_scale - 1.0) * (v_cond - v_uncond_modality)
        )

        # 2. CFG Standard Deviation Rescaling
        if self.params.rescale_scale > 0.0:
            std_cond = v_cond.std(dim=(1, 2), keepdim=True) + 1e-8
            std_guided = v_guided.std(dim=(1, 2), keepdim=True) + 1e-8
            factor = std_cond / std_guided
            factor = self.params.rescale_scale * factor + (1.0 - self.params.rescale_scale)
            v_guided = v_guided * factor

        return v_guided

Empirical Benchmark Evaluation

Quantitative evaluation comparing joint audio-video synchronization, visual quality, and inference performance:

Troubleshooting Common LTX-2 Deployment Faults

1. Acoustic Drift and Lip-Sync Phase Disconnect

• Symptom: Generated dialogue or sound effects precede or lag corresponding video actions by $200\text{–}500\text{ ms}$.
• Root Cause: modality_scale set too low ($\le 1.0$), or audio sampling rates resampled incorrectly before entering the mel frontend.
• Remedy: Increase modality_scale to $3.0$ in MultiModalGuiderParams and ensure raw audio inputs are strictly resampled to $16\text{ kHz}$ mono before VAE encoding.

2. Video VAE Tensor Dimension Assert Failure

• Symptom: AssertionError: Video frame dimension must satisfy 8k + 1.
• Root Cause: Supplying frame counts not matching the $F=8k+1$ temporal contract (e.g., passing $120$ instead of $121$).
• Remedy: Snap input video frame counts via: num_frames = ((num_frames - 1) // 8) * 8 + 1.

3. FP8 Scaled Matrix Multiplication NaN Spikes on Hopper

• Symptom: Loss evaluates to NaN or generated video frames corrupt into grey artifacts when running fp8-scaled-mm.
• Root Cause: Incompatible cuBLAS gemm scalar scale factor calibrations across intermediate cross-attention buffers.
• Remedy: Switch quantization policy from fp8-scaled-mm to fp8-cast for stable on-the-fly dynamic range upcasting.

References

1. Lightricks Research. (2025). LTX-2: Asymmetric Dual-Stream Audio-Video Diffusion Transformer. Technical Report.
2. Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nicklas, M., & Le, M. (2023). Flow Matching for Generative Modeling. ICLR 2023.
3. Zhou, Y., et al. (2024). Spatio-Temporal Guidance for High-Fidelity Video Diffusion. arXiv:2411.18640.
4. Lin, S., et al. (2024). Common Diffusion Noise Schedules and Sample Steps are Flawed. WACV 2024.
5. Chen, J., et al. (2023). PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. arXiv:2310.00426.