4D Gaussian Splatting (4DGS): HexPlane Spatiotemporal Neural Fields and Dynamic Splatting

Dynamic Novel View Synthesis: Implicit Volume Sampling vs. Spatiotemporal Splatting

Dynamic Novel View Synthesis (Dynamic NVS) models non-rigid geometry, articulated motion, and time-varying radiance fields from multi-view video sequences. Prior methods based on Dynamic NeRF (e.g., D-NeRF, HyperNeRF, TiNeuVox) parameterized scenes via time-conditioned coordinate networks:

$F_{\theta }:(\mathbf{x},t,\mathbf{d})\mapsto (\mathbf{c},\sigma ,\Delta \mathbf{x})$

While continuous implicit fields model continuous topological transformations, volumetric ray-marching requires querying coordinate MLPs hundreds of times per ray. Consequently, inference latency remains bounded at sub-real-time rates ($\le 5\text{ FPS}$ at $1080\text{p}$).

4D Gaussian Splatting (4DGS) (Wu et al., CVPR 2024; Yang et al., 2024) extends explicit 3D Gaussian representations into the temporal domain by combining a canonical 3D Gaussian cloud with spatiotemporal neural deformation fields (HexPlane decomposition or coordinate MLPs). By evaluating deformation offsets in parallel and rasterizing deformed primitives through tile-based differentiable splatting, 4DGS delivers $\ge 80\text{–}150\text{ FPS}$ at $1080\text{p}$ while cutting training time from tens of hours to under 45 minutes.

Architectural Comparison

Spatiotemporal Mathematical Formulations

Figure 1: Complete forward deformation and gradient backpropagation dataflow of 4D Gaussian Splatting.

1. The Canonical Gaussian Primitive Set

The dynamic scene is anchored by a canonical set of $N$ explicit 3D Gaussians:

${\mathcal{S}}_{\text{can}}={\left\{{\mathcal{G}}_i^{\text{can}}=\left({\mathbf{\mu }}_i^{\text{can}},{\mathbf{q}}_i^{\text{can}},{\mathbf{s}}_i^{\text{can}},{\alpha }_i^{\text{can}},{\mathbf{k}}_i^{\text{can}}\right)\right\}}_{i=1}^N$

where ${\mathbf{\mu }}_i^{\text{can}}\in {\mathbb{R}}^3$ denotes center coordinates, ${\mathbf{q}}_i^{\text{can}}\in \mathbb{H}$ represents the unit rotation quaternion, ${\mathbf{s}}_i^{\text{can}}\in {\mathbb{R}}^3$ is the log-scale vector, ${\alpha }_i^{\text{can}}\in [0,1]$ is the base opacity, and ${\mathbf{k}}_i^{\text{can}}\in {\mathbb{R}}^{K\times 3}$ contains Spherical Harmonics coefficients.

2. HexPlane Spatiotemporal Decomposition

Directly parameterizing a 4D dense grid ${\mathbb{R}}^{X\times Y\times Z\times T}$ suffers from exponential $O(N^4)$ memory scaling. 4DGS resolves this by decomposing the $4\text{D}$ space-time volume into six orthogonal 2D feature planes $\Pi =\{\text{XY},\text{XZ},\text{YZ},\text{XT},\text{YT},\text{ZT}\}$:

$\mathbf{f}(\mathbf{x},t)={⨁}_{\pi \in \Pi }\text{BilinearSample}\left(M_{\pi },\pi (\mathbf{x},t)\right)$

where $\pi (\mathbf{x},t)$ projects 4D coordinates onto corresponding 2D sub-spaces (e.g., ${\pi }_{\text{XT}}(\mathbf{x},t)=(x,t)$), $M_{\pi }\in {\mathbb{R}}^{C\times H_{\pi }\times W_{\pi }}$ is a learnable multi-resolution feature grid, and $⨁$ denotes channel concatenation followed by linear mixing.

The spatial planes ($M_{\text{XY}},M_{\text{XZ}},M_{\text{YZ}}$) preserve high-frequency static geometry, while the temporal planes ($M_{\text{XT}},M_{\text{YT}},M_{\text{ZT}}$) capture continuous dynamic motion trajectories along principal spatial axes.

3. Neural Deformation & Attribute Transformation

A lightweight Multi-Layer Perceptron (MLP) ${\Phi }_{\theta }$ maps spatiotemporal features $\mathbf{f}({\mathbf{\mu }}_i^{\text{can}},t)$ to time-varying geometric and appearance offsets:

$(\Delta {\mathbf{\mu }}_i,\Delta {\mathbf{q}}_i,\Delta {\mathbf{s}}_i,\Delta {\alpha }_i,\Delta {\mathbf{k}}_i)={\Phi }_{\theta }\left(\mathbf{f}({\mathbf{\mu }}_i^{\text{can}},t)\right)$

The primitive parameters at timestamp $t$ are computed via:

${\mathbf{\mu }}_i(t)={\mathbf{\mu }}_i^{\text{can}}+\Delta {\mathbf{\mu }}_i(t)$

${\mathbf{q}}_i(t)=\frac{{\mathbf{q}}_i^{\text{can}}\otimes \Delta {\mathbf{q}}_i(t)}{\Vert {\mathbf{q}}_i^{\text{can}}\otimes \Delta {\mathbf{q}}_i(t){\Vert }_2}$

${\mathbf{s}}_i(t)={\mathbf{s}}_i^{\text{can}}\odot \exp (\Delta {\mathbf{s}}_i(t))$

${\alpha }_i(t)=\sigma \left({\sigma }^{-1}({\alpha }_i^{\text{can}})+\Delta {\alpha }_i(t)\right)$

where $\otimes$ denotes quaternion Hamilton product, $\odot$ represents element-wise multiplication, and $\sigma (\cdot )$ is the sigmoid activation.

4. Spatiotemporal Composite Loss Function

Training optimizes canonical parameters ${\mathcal{S}}_{\text{can}}$, feature planes $\{M_{\pi }\}$, and decoder weights ${\Phi }_{\theta }$ under a joint objective:

$\mathcal{L}={\mathcal{L}}_{\text{photo}}+{\lambda }_{\text{smooth}}{\mathcal{L}}_{\text{smooth}}+{\lambda }_{\text{iso}}{\mathcal{L}}_{\text{iso}}+{\lambda }_{\text{TV}}{\mathcal{L}}_{\text{TV}}$

4.1 Photometric Reconstruction Loss

${\mathcal{L}}_{\text{photo}}=(1-\beta )\Vert {\hat{I}}_t-I_t{\Vert }_1+\beta \left(1-\text{SSIM}({\hat{I}}_t,I_t)\right),\beta =0.2$

4.2 Temporal Smoothness Regularization

Prevents high-frequency temporal flickering (“boiling” artifacts) by penalizing acceleration and velocity spikes: ${\mathcal{L}}_{\text{smooth}}=\frac{1}{N}{\sum }_{i=1}^N\left({\Vert \frac{\partial {\mathbf{\mu }}_i(t)}{\partial t}\Vert }_2^2+{\Vert \frac{{\partial }^2{\mathbf{\mu }}_i(t)}{\partial t^2}\Vert }_2^2\right)$

4.3 As-Rigid-As-Possible (ARAP) Local Rigidity Loss

Preserves local manifold topology across $k$-nearest canonical neighbors $\mathcal{N}(i)$: ${\mathcal{L}}_{\text{iso}}=\frac{1}{N|\mathcal{N}(i)|}{\sum }_{i=1}^N{\sum }_{j\in \mathcal{N}(i)}{\left(\Vert {\mathbf{\mu }}_i(t)-{\mathbf{\mu }}_j(t){\Vert }_2-\Vert {\mathbf{\mu }}_i^{\text{can}}-{\mathbf{\mu }}_j^{\text{can}}{\Vert }_2\right)}^2$

4.4 Total Variation (TV) Regularization on Temporal Feature Planes

Enforces spatial and temporal continuity across feature plane grids: ${\mathcal{L}}_{\text{TV}}={\sum }_{\pi \in \{\text{XT, YT, ZT}\}}\frac{1}{H_{\pi }{W}_{\pi }}{\sum }_{u,v}\left(\Vert M_{\pi }[u+1,v]-M_{\pi }[u,v]{\Vert }_2^2+\Vert M_{\pi }[u,v+1]-M_{\pi }[u,v]{\Vert }_2^2\right)$

Production PyTorch Implementation

Below is a complete, modular PyTorch implementation of the HexPlane Spatiotemporal Feature Field and Deformation Decoder.

Step 1: Multi-Resolution HexPlane Feature Field

from __future__ import annotations

import torch
import torch.nn as nn
import torch.nn.functional as F


class HexPlaneField(nn.Module):
    def __init__(
        self,
        feature_dim: int = 32,
        spatial_res: int = 256,
        temporal_res: int = 64,
    ) -> None:
        super().__init__()
        self.feature_dim = feature_dim
        
        # 3 Spatial Planes (XY, XZ, YZ)
        self.plane_xy = nn.Parameter(torch.randn(1, feature_dim, spatial_res, spatial_res) * 0.01)
        self.plane_xz = nn.Parameter(torch.randn(1, feature_dim, spatial_res, spatial_res) * 0.01)
        self.plane_yz = nn.Parameter(torch.randn(1, feature_dim, spatial_res, spatial_res) * 0.01)

        # 3 Spatiotemporal Planes (XT, YT, ZT)
        self.plane_xt = nn.Parameter(torch.randn(1, feature_dim, spatial_res, temporal_res) * 0.01)
        self.plane_yt = nn.Parameter(torch.randn(1, feature_dim, spatial_res, temporal_res) * 0.01)
        self.plane_zt = nn.Parameter(torch.randn(1, feature_dim, spatial_res, temporal_res) * 0.01)

    def sample_plane(self, plane: torch.Tensor, coords: torch.Tensor) -> torch.Tensor:
        """Bilinear interpolation on 2D plane with normalized coords in [-1, 1]."""
        # coords shape: [1, 1, N, 2] -> grid_sample expects [B, H_out, W_out, 2]
        grid = coords.unsqueeze(0).unsqueeze(0)
        sampled = F.grid_sample(plane, grid, mode="bilinear", padding_mode="border", align_corners=True)
        return sampled.squeeze(0).squeeze(1).transpose(0, 1) # [N, feature_dim]

    def forward(self, xyz: torch.Tensor, t: torch.Tensor) -> torch.Tensor:
        """
        xyz: [N, 3] normalized to [-1, 1]
        t: [N, 1] normalized to [-1, 1]
        """
        x, y, z = xyz[:, 0:1], xyz[:, 1:2], xyz[:, 2:3]

        coords_xy = torch.cat([x, y], dim=-1)
        coords_xz = torch.cat([x, z], dim=-1)
        coords_yz = torch.cat([y, z], dim=-1)

        coords_xt = torch.cat([x, t], dim=-1)
        coords_yt = torch.cat([y, t], dim=-1)
        coords_zt = torch.cat([z, t], dim=-1)

        feat_xy = self.sample_plane(self.plane_xy, coords_xy)
        feat_xz = self.sample_plane(self.plane_xz, coords_xz)
        feat_yz = self.sample_plane(self.plane_yz, coords_yz)

        feat_xt = self.sample_plane(self.plane_xt, coords_xt)
        feat_yt = self.sample_plane(self.plane_yt, coords_yt)
        feat_zt = self.sample_plane(self.plane_zt, coords_zt)

        # Concatenate spatial and temporal features -> [N, feature_dim * 6]
        return torch.cat([feat_xy, feat_xz, feat_yz, feat_xt, feat_yt, feat_zt], dim=-1)

Step 2: Neural Deformation Decoder

from __future__ import annotations

import torch
import torch.nn as nn


class DeformationDecoder(nn.Module):
    def __init__(self, in_features: int = 192, hidden_dim: int = 128) -> None:
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_features, hidden_dim),
            nn.SiLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.SiLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.SiLU(),
        )

        # Dedicated linear heads for geometric and opacity offsets
        self.head_dx = nn.Linear(hidden_dim, 3)     # Delta mu
        self.head_dq = nn.Linear(hidden_dim, 4)     # Delta quaternion
        self.head_ds = nn.Linear(hidden_dim, 3)     # Delta scale
        self.head_dalpha = nn.Linear(hidden_dim, 1) # Delta opacity

        # Zero-initialize output projection layers so initial state matches canonical
        nn.init.zeros_(self.head_dx.weight)
        nn.init.zeros_(self.head_dx.bias)
        nn.init.zeros_(self.head_dq.weight)
        nn.init.zeros_(self.head_dq.bias)
        nn.init.zeros_(self.head_ds.weight)
        nn.init.zeros_(self.head_ds.bias)
        nn.init.zeros_(self.head_dalpha.weight)
        nn.init.zeros_(self.head_dalpha.bias)

    def forward(self, features: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
        h = self.net(features)
        delta_x = self.head_dx(h)
        delta_q = self.head_dq(h)
        delta_s = self.head_ds(h)
        delta_alpha = self.head_dalpha(h)
        return delta_x, delta_q, delta_s, delta_alpha

Step 3: Deformed Primitive Assembly & Loss Optimization

from __future__ import annotations

import torch
from deformation_decoder import DeformationDecoder
from hexplane import HexPlaneField


def compute_tv_loss(planes: list[torch.Tensor]) -> torch.Tensor:
    tv_loss = 0.0
    for plane in planes:
        dh = torch.abs(plane[:, :, 1:, :] - plane[:, :, :-1, :]).mean()
        dw = torch.abs(plane[:, :, :, 1:] - plane[:, :, :, :-1]).mean()
        tv_loss = tv_loss + (dh + dw)
    return tv_loss


def dynamic_forward_step(
    canonical_xyz: torch.Tensor,
    canonical_rot: torch.Tensor,
    canonical_scale: torch.Tensor,
    canonical_opacity: torch.Tensor,
    timestamp: float,
    hexplane: HexPlaneField,
    decoder: DeformationDecoder,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
    N = canonical_xyz.shape[0]
    t_tensor = torch.full((N, 1), timestamp, device=canonical_xyz.device, dtype=torch.float32)

    # 1. Sample Spatiotemporal Features
    spatiotemporal_feats = hexplane(canonical_xyz, t_tensor)

    # 2. Decode Deformation Offsets
    dx, dq, ds, dalpha = decoder(spatiotemporal_feats)

    # 3. Apply Offsets
    deformed_xyz = canonical_xyz + dx
    deformed_rot = F.normalize(canonical_rot + dq, dim=-1)
    deformed_scale = canonical_scale * torch.exp(ds)
    deformed_opacity = torch.sigmoid(torch.logit(canonical_opacity.clamp(1e-4, 1.0 - 1e-4)) + dalpha)

    return deformed_xyz, deformed_rot, deformed_scale, deformed_opacity

Empirical Benchmark Evaluation

Quantitative evaluation across the standard Plenoptic Video Dataset and D-NeRF Synthetic Benchmark:

Troubleshooting Common Dynamic Synthesis Faults

1. High-Frequency Temporal Jitter (“Boiling” Artifacts)

• Symptom: Subtle per-frame texture flickering across flat, smooth surfaces.
• Root Cause: Overfitting in the temporal deformation MLP due to unconstrained high-frequency temporal gradient steps.
• Remedy: Increase ${\lambda }_{\text{smooth}}$ by $3\times$, enforce Total Variation regularization on $M_{\text{XT}},M_{\text{YT}},M_{\text{ZT}}$, and initialize deformation decoder heads to zero.

2. Static Background Deformation Leakage

• Symptom: Rigid background elements (floors, walls) subtly warp as foreground objects move.
• Root Cause: The deformation network applies non-zero offsets globally without spatial motion gating.
• Remedy: Compute temporal variance across training frames to mask static background Gaussians, freezing their positions to $\Delta \mathbf{\mu }=0$.

3. Rapid Motion Ghosting Under Sparse Camera Coverage

• Symptom: Trailing semi-transparent ghosts during fast movements (>50 px/frame).
• Root Cause: Photometric loss alone fails to disambiguate large inter-frame displacements.
• Remedy: Integrate 2D optical flow supervision (RAFT / SEA-RAFT) with ${\mathcal{L}}_{\text{flow}}$ and apply local rigidity ARAP regularization ${\mathcal{L}}_{\text{iso}}$.

References

1. Wu, G., Yi, T., Fang, J., Xie, L., Zhang, X., Wei, W., Liu, W., Tian, Q., & Wang, X. (2024). 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering. IEEE/CVF CVPR 2024.
2. Yang, Z., Gao, X., Zhou, W., Sha, S., Sheng, L., & Fan, W. (2024). Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction. CVPR.
3. Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics (TOG).
4. Fridovich-Keil, S., Meanti, G., Warburg, F. R., Recht, B., & Kanazawa, A. (2023). K-Planes: Explicit Radiance Fields in Space, Time, and Appearance. CVPR.
5. Guo, Y., Wang, K., Kang, W., Yang, S., & Fang, Y. (2024). STAG4D: Spatial-Temporal Anchored Gaussian Splatting for 4D Reconstruction. arXiv:2404.04394.