NeRF: Neural Radiance Fields, Differentiable Volume Rendering, and PyTorch Architecture

Continuous 5D Implicit Scene Representation

Classical 3D scene reconstruction paradigms rely on explicit geometric representations: textured polygonal meshes, dense point clouds, or discretized volumetric occupancy grids. These formulations present severe limitations when modeling complex non-Lambertian view-dependent lighting (specular reflections, Fresnel highlights), semi-transparent media (glass, smoke, water), and thin geometric topologies (foliage, hair, wireframes).

Neural Radiance Fields (NeRF) (Mildenhall et al., ECCV 2020) replaces discrete geometric data structures with a continuous implicit 5D vector-valued function parameterized by a fully connected Multilayer Perceptron (MLP) $F_{\Theta }$:

$F_{\Theta }:(\mathbf{x},\mathbf{d})⟼(\mathbf{c},\sigma )$

where:

• $\mathbf{x}=(x,y,z)\in {\mathbb{R}}^3$ represents spatial 3D world coordinates.
• $\mathbf{d}=(\theta ,\varphi )\in {\mathbb{S}}^2$ (or normalized 3D unit vector $\mathbf{d}\in {\mathbb{R}}^3$) represents the viewing ray direction.
• $\mathbf{c}=(r,g,b)\in [0,1{]}^3$ denotes emitted directional radiance.
• $\sigma \in {\mathbb{R}}_{\ge 0}$ denotes differential volume density (optical extinction coefficient).

By integrating the radiative transfer equation along camera rays, NeRF optimizes the MLP directly against 2D ground-truth photographs via differentiable volume rendering, requiring no 3D geometric supervision.

3D Scene Representations Comparison

Mathematical Foundations

Figure 1: NeRF forward computation pipeline decomposing 5D ray coordinates into view-independent density $\sigma$ and view-dependent radiance $\mathbf{c}$ via differentiable volume rendering.

1. The Continuous Radiative Transfer Equation

Under the emission-absorption physical optical model (no secondary scattering or internal participating light emission), the expected radiance $C(\mathbf{r})$ along a camera ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ parameterized between bounds $[t_n,t_f]$ is expressed analytically as:

$C(\mathbf{r})={\int }_{t_n}^{t_f}T(t)\sigma (\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),\mathbf{d})\mathrm{d}t$

where the accumulated transmittance $T(t)$ represents the continuous probability that the optical photon traverses the medium from $t_n$ to $t$ without absorption:

$T(t)=\exp \left(-{\int }_{t_n}^t\sigma (\mathbf{r}(s))\mathrm{d}s\right)$

2. Numerical Quadrature and Discrete Alpha Compositing

Because continuous integration along an implicit MLP cannot be computed in closed form, NeRF applies stratified sampling to partition $[t_n,t_f]$ into $N$ evenly-spaced intervals $t_i\sim \mathcal{U}\left[t_n+\frac{i-1}{N}(t_f-t_n),t_n+\frac{i}{N}(t_f-t_n)\right]$.

Evaluating the integral via numerical quadrature yields the deterministic discrete alpha-compositing equation (Max, 1995):

$\hat{C}(\mathbf{r})={\sum }_{i=1}^N{T}_i{\alpha }_i{\mathbf{c}}_i={\sum }_{i=1}^N{w}_i{\mathbf{c}}_i$

where the individual sample opacity ${\alpha }_i$ across step interval ${\delta }_i=t_{i+1}-t_i$ is derived from the Beer-Lambert extinction law:

${\alpha }_i=1-\exp (-{\sigma }_i{\delta }_i)$

and the discrete transmittance $T_i$ represents cumulative non-absorption over preceding intervals:

$T_i={\prod }_{j=1}^{i-1}(1-{\alpha }_j)=\exp \left(-{\sum }_{j=1}^{i-1}{\sigma }_j{\delta }_j\right)$

The composite weight $w_i=T_i{\alpha }_i$ satisfies ${\sum }_{i=1}^N{w}_i\le 1.0$, defining a discrete probability density function along the ray.

3. Fourier Positional Encoding and Spectral Bias

Standard fully-connected networks with ReLU activations suffer from spectral bias (Rahaman et al., ICML 2019), exhibiting an inductive preference toward learning low-frequency functions and blurring high-frequency spatial textures and sharp edges.

NeRF remaps low-dimensional inputs $\mathbf{p}\in {\mathbb{R}}^d$ into a high-dimensional Fourier feature space using positional encoding $\gamma :\mathbb{R}\to {\mathbb{R}}^{2L}$:

$\gamma (p)=\left(\sin \left(2^0\pi p\right),\cos \left(2^0\pi p\right),\sin \left(2^1\pi p\right),\cos \left(2^1\pi p\right),\dots ,\sin \left(2^{L-1}\pi p\right),\cos \left(2^{L-1}\pi p\right)\right)$

For spatial coordinates $\mathbf{x}\in {\mathbb{R}}^3$, setting $L_{\text{pos}}=10$ produces an input vector of dimension $3\times 2\times 10=60$ ($63$ when including raw coordinates). For ray direction $\mathbf{d}\in {\mathbb{S}}^2$, setting $L_{\text{dir}}=4$ yields dimension $3\times 2\times 4=24$ ($27$ with raw coordinates).

Neural Tangent Kernel (NTK) Interpretation:

Tancik et al. (NeurIPS 2020) demonstrated that Fourier mapping transforms the stationary NTK of the coordinate MLP from a slow-decaying kernel into a tuned shift-invariant Gaussian-like kernel, enabling the network to fit spatial frequencies up to bandwidth $2^{L-1}\pi$.

4. Hierarchical Importance Sampling

Uniformly distributing $N$ samples across $[t_n,t_f]$ wastes computational budget on unoccupied free space and occluded regions behind opaque surfaces. NeRF utilizes a coarse-to-fine hierarchical sampling strategy:

1. Coarse Stage: $N_c=64$ samples are drawn uniformly to compute coarse weights:

${\hat{w}}_i=\frac{w_i}{{\sum }_{j=1}^{N_c}{w}_j+ϵ}$

2. Fine Stage: Treating ${\hat{w}}_i$ as a piecewise-constant probability density function (PDF), $N_f=128$ additional samples are drawn via inverse CDF transform sampling:

$u\sim \mathcal{U}[0,1)⟹t_{\text{fine}}=F^{-1}(u)$

3. Composite Evaluation: The combined set of $N_c+N_f=192$ samples is sorted along the ray and evaluated by the fine MLP $F_{{\Theta }_f}$ to compute the final rendered color ${\hat{C}}_f(\mathbf{r})$.

5. Joint Optimization Loss Objective

The network parameters $\Theta =\{{\Theta }_c,{\Theta }_f\}$ are optimized end-to-end via photometric Mean Squared Error (MSE) across randomly sampled ray batches $\mathcal{R}$:

${\mathcal{L}}_{\text{Total}}(\Theta )={\sum }_{\mathbf{r}\in \mathcal{R}}\left[{\Vert {\hat{C}}_c(\mathbf{r})-C_{\text{gt}}(\mathbf{r})\Vert }_2^2+{\Vert {\hat{C}}_f(\mathbf{r})-C_{\text{gt}}(\mathbf{r})\Vert }_2^2\right]$

Supervising the coarse network alongside the fine network ensures that the coarse density distribution accurately guides fine importance sampling throughout training.

Production PyTorch Implementation

Below is a complete, modular PyTorch implementation of the NeRF network, the vectorized differentiable volume renderer, and the inverse-CDF hierarchical importance sampler.

Step 1: Positional Encoding and NeRF MLP Architecture

from __future__ import annotations

import torch
import torch.nn as nn
import torch.nn.functional as F


class FourierPositionalEncoding(nn.Module):
    """Sinusoidal positional encoding mapping x in R^D to R^(D * 2 * L)."""

    def __init__(self, num_freqs: int, include_input: bool = True) -> None:
        super().__init__()
        self.num_freqs = num_freqs
        self.include_input = include_input
        self.freq_bands = 2.0 ** torch.linspace(0.0, num_freqs - 1, num_freqs)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        out = [x] if self.include_input else []
        for freq in self.freq_bands.to(x.device):
            out.append(torch.sin(freq * torch.pi * x))
            out.append(torch.cos(freq * torch.pi * x))
        return torch.cat(out, dim=-1)


class NeRFMLP(nn.Module):
    """Canonical 8-layer NeRF MLP with view-independent density and view-dependent color."""

    def __init__(
        self,
        d_pos: int = 3,
        d_dir: int = 3,
        l_pos: int = 10,
        l_dir: int = 4,
        net_width: int = 256,
        skip_layers: tuple[int, ...] = (4,),
    ) -> None:
        super().__init__()
        self.skip_layers = skip_layers
        self.pe_pos = FourierPositionalEncoding(l_pos, include_input=True)
        self.pe_dir = FourierPositionalEncoding(l_dir, include_input=True)

        in_dim_pos = d_pos + 2 * d_pos * l_pos # 63
        in_dim_dir = d_dir + 2 * d_dir * l_dir # 27

        # 1. Density Trunk (8 layers)
        self.pts_linears = nn.ModuleList([nn.Linear(in_dim_pos, net_width)])
        for i in range(1, 8):
            in_features = in_dim_pos + net_width if i in skip_layers else net_width
            self.pts_linears.append(nn.Linear(in_features, net_width))

        # 2. Output Heads
        self.density_linear = nn.Linear(net_width, 1)
        self.feature_linear = nn.Linear(net_width, net_width)
        self.color_linear = nn.Sequential(
            nn.Linear(net_width + in_dim_dir, net_width // 2),
            nn.ReLU(),
            nn.Linear(net_width // 2, 3),
            nn.Sigmoid(),
        )

    def forward(self, x: torch.Tensor, d: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
        # x: [N_rays, N_samples, 3], d: [N_rays, N_samples, 3]
        x_encoded = self.pe_pos(x)
        h = x_encoded

        for i, layer in enumerate(self.pts_linears):
            h = F.relu(layer(h))
            if i in self.skip_layers:
                h = torch.cat([x_encoded, h], dim=-1)

        # Density is strictly non-negative (ReLU) and view-independent
        sigma = F.relu(self.density_linear(h))
        feature = self.feature_linear(h)

        # Color is conditioned on both spatial feature and viewing direction
        d_encoded = self.pe_dir(d)
        rgb = self.color_linear(torch.cat([feature, d_encoded], dim=-1))

        return rgb, sigma

Step 2: Differentiable Volume Renderer and Hierarchical Sampler

from __future__ import annotations

import torch


def volume_render_quadrature(
    rgb: torch.Tensor,
    sigma: torch.Tensor,
    t_vals: torch.Tensor,
    white_background: bool = False,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
    """Computes discrete alpha compositing along sampled ray intervals."""
    # Distance intervals: delta_i = t_(i+1) - t_i
    dists = t_vals[..., 1:] - t_vals[..., :-1]
    dists = torch.cat([dists, torch.full_like(dists[..., :1], 1e10)], dim=-1)

    # Individual opacities: alpha_i = 1 - exp(-sigma_i * delta_i)
    alpha = 1.0 - torch.exp(-sigma.squeeze(-1) * dists)

    # Transmittance: T_i = prod_(j < i) (1 - alpha_j)
    trans = torch.cumprod(
        torch.cat([torch.ones_like(alpha[..., :1]), 1.0 - alpha + 1e-10], dim=-1),
        dim=-1,
    )[..., :-1]

    # Composite weights: w_i = T_i * alpha_i
    weights = alpha * trans

    # Accumulated maps
    rgb_map = torch.sum(weights.unsqueeze(-1) * rgb, dim=-2)
    depth_map = torch.sum(weights * t_vals, dim=-1)
    acc_map = torch.sum(weights, dim=-1)

    if white_background:
        rgb_map = rgb_map + (1.0 - acc_map.unsqueeze(-1))

    return rgb_map, depth_map, acc_map, weights


def sample_pdf_inverse_cdf(
    t_coarse: torch.Tensor,
    weights: torch.Tensor,
    num_fine_samples: int,
) -> torch.Tensor:
    """Draws fine importance samples from the normalized coarse PDF."""
    pdf = (weights + 1e-5) / torch.sum(weights + 1e-5, dim=-1, keepdim=True)
    cdf = torch.cumsum(pdf, dim=-1)
    cdf = torch.cat([torch.zeros_like(cdf[..., :1]), cdf], dim=-1) # [N_rays, N_c + 1]

    # Uniform random points
    u = torch.rand(*cdf.shape[:-1], num_fine_samples, device=cdf.device).contiguous()

    # Invert CDF via searchsorted
    inds = torch.searchsorted(cdf, u, right=True)
    below = torch.clamp(inds - 1, min=0)
    above = torch.clamp(inds, max=cdf.shape[-1] - 1)
    inds_g = torch.stack([below, above], dim=-1)

    matched_shape = (*inds_g.shape[:-2], inds_g.shape[-2], cdf.shape[-1])
    cdf_g = torch.gather(cdf.unsqueeze(-2).expand(matched_shape), -1, inds_g)
    t_g = torch.gather(t_coarse.unsqueeze(-2).expand(matched_shape), -1, inds_g)

    denom = cdf_g[..., 1] - cdf_g[..., 0]
    denom = torch.where(denom < 1e-5, torch.ones_like(denom), denom)
    t_fine = t_g[..., 0] + (u - cdf_g[..., 0]) / denom * (t_g[..., 1] - t_g[..., 0])

    return t_fine.detach()

Step 3: End-to-End Hierarchical Ray-Marching Pipeline

from __future__ import annotations

import torch
import torch.nn as nn
from nerf_model import NeRFMLP
from volume_renderer import sample_pdf_inverse_cdf, volume_render_quadrature


class NeRFRenderEngine(nn.Module):
    """End-to-end two-stage coarse-fine NeRF ray tracing pipeline."""

    def __init__(self, coarse_model: NeRFMLP, fine_model: NeRFMLP) -> None:
        super().__init__()
        self.coarse_model = coarse_model
        self.fine_model = fine_model

    def forward(
        self,
        rays_o: torch.Tensor,
        rays_d: torch.Tensor,
        near: float,
        far: float,
        num_coarse: int = 64,
        num_fine: int = 128,
        white_background: bool = False,
    ) -> dict[str, torch.Tensor]:
        num_rays = rays_o.shape[0]

        # 1. Stratified coarse sampling
        t_vals = torch.linspace(0.0, 1.0, num_coarse, device=rays_o.device)
        t_coarse = near + (far - near) * (t_vals + torch.rand(num_rays, num_coarse, device=rays_o.device) / num_coarse)

        pts_coarse = rays_o.unsqueeze(1) + rays_d.unsqueeze(1) * t_coarse.unsqueeze(-1)
        dirs_coarse = rays_d.unsqueeze(1).expand_as(pts_coarse)

        # 2. Coarse MLP pass & rendering
        rgb_c, sigma_c = self.coarse_model(pts_coarse, dirs_coarse)
        rgb_map_c, depth_c, acc_c, weights_c = volume_render_quadrature(
            rgb_c, sigma_c, t_coarse, white_background=white_background
        )

        # 3. Hierarchical fine sampling
        t_fine = sample_pdf_inverse_cdf(t_coarse, weights_c, num_fine)
        t_all, _ = torch.sort(torch.cat([t_coarse, t_fine], dim=-1), dim=-1)

        pts_fine = rays_o.unsqueeze(1) + rays_d.unsqueeze(1) * t_all.unsqueeze(-1)
        dirs_fine = rays_d.unsqueeze(1).expand_as(pts_fine)

        # 4. Fine MLP pass & rendering
        rgb_f, sigma_f = self.fine_model(pts_fine, dirs_fine)
        rgb_map_f, depth_f, acc_f, _ = volume_render_quadrature(
            rgb_f, sigma_f, t_all, white_background=white_background
        )

        return {
            "rgb_coarse": rgb_map_c,
            "rgb_fine": rgb_map_f,
            "depth_fine": depth_f,
            "acc_fine": acc_f,
        }

Empirical Benchmark Evaluation

Quantitative reconstruction fidelity and inference speed across canonical computer vision datasets:

Troubleshooting Common NeRF Training Faults

1. “Foggy” Cloudy Floater Artifacts in Empty Space

• Symptom: Translucent noise clouds float in foreground regions with low ray intersection density.
• Root Cause: Unconstrained density updates in regions unconstrained by multi-view ray intersections; absence of background regularization.
• Remedy: Incorporate distortion loss ${\mathcal{L}}_{\text{dist}}$ (Mip-NeRF 360) or add an opacity penalty term: ${\mathcal{L}}_{\text{reg}}=\lambda {\sum }_i|w_i|$.

2. Camera Coordinate Frame Handedness Mismatch

• Symptom: Training loss oscillates around high MSE values ($\text{PSNR}<12\text{ dB}$); rendered images display grey uniform patterns.
• Root Cause: COLMAP outputs OpenCV convention ($+X\text{ right},+Y\text{ down},+Z\text{ forward}$), whereas standard NeRF renderers expect OpenGL convention ($+X\text{ right},+Y\text{ up},-Z\text{ forward}$).
• Remedy: Transform camera extrinsic matrices by flipping $Y$ and $Z$ column vectors: ${\mathbf{E}}_{\text{OpenGL}}={\mathbf{E}}_{\text{OpenCV}}\cdot \mathrm{diag}(1,-1,-1,1)$.

3. View-Dependent Radiance Overfitting (Specular Artifacts)

• Symptom: Sharp novel views display flickering white spots; specular reflections bake into wrong spatial depths.
• Root Cause: Positional encoding degree for direction ($L_{\text{dir}}$) set too high relative to spatial resolution, allowing the color head to bypass geometric density constraints.
• Remedy: Limit $L_{\text{dir}}\le 4$, restrict the color head width to $128$ units, and delay direction input injection until the final linear projection layer.

References

1. Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. European Conference on Computer Vision (ECCV 2020).
2. Müller, T., Evans, A., Schied, C., & Keller, A. (2022). Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics (SIGGRAPH 2022).
3. Barron, J. T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., & Srinivasan, P. P. (2021). Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. International Conference on Computer Vision (ICCV 2021).
4. Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics (SIGGRAPH 2023).
5. Tancik, M., et al. (2020). Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. Advances in Neural Information Processing Systems (NeurIPS 2020).
6. Max, N. (1995). Optical Models for Direct Volume Rendering. IEEE Transactions on Visualization and Computer Graphics.