Introduction to Neural Rendering

Overview

• Presentation introduces “neural rendering” and practical paths for using ML in graphics:

  1. Post-process neural upscaling/denoising (e.g., DLSS).
  2. Neural components embedded inside the rendering pipeline (material decoders, texture decoders).
  3. Largely generative neural pipelines that replace parts of the traditional renderer.

• Talk structure:

  • Background and real‑time constraints (Shannon).
  • Three case studies (Alexey/Alexi): neural texture compression (NTC), neural materials, and Omniverse NeuralRec for autonomous vehicle (AV) simulation.
  • Wrap-up and tooling.

---

Core technical points and real‑time constraints

• Trend: moving from hand‑coded analytic models (BRDFs, BCn texture compression) to learned representations that trade explicit equations for data‑driven models able to capture complex non‑linear phenomena.

• Real‑time constraints when putting ML inside the render loop:

  • Pixel‑rate inference requires tiny, fused MLPs that run on‑chip to avoid expensive memory round trips.
  • Use cooperative vector/matrix ops and tensor cores for throughput.
  • Maintain a single codebase: the same implementation should be used for training (Python/PyTorch) and deployment (C++/shaders) to avoid divergence.

• Slang + SlangPy (solution to these constraints):

  • First‑class automatic differentiation: mark functions differentiable and let the compiler generate the backward pass; custom derivatives can be provided.
  • Compilation targets: CUDA / HLSL / GLSL, with cooperative matrix/vector APIs.
  • Python bindings (SlangPy) let Slang code be invoked and differentiated inside training loops so the same source runs in training and runtime.

Key constraint: tiny, fused networks that minimize memory traffic and leverage hardware cooperative operations are required for pixel‑rate neural components in real‑time rendering.

---

Case study 1 — Neural Texture Compression (NTC)

Concept

Replace full‑resolution color texels with latent feature maps (latent textures). A compact decoder MLP reconstructs texel colors on demand.

Key techniques

• Positional encoding of UVs to capture high‑frequency detail.
• Deterministic reconstruction (same latent + weights → identical output).
• Training optimizes decoder weights and latent codes against ground‑truth textures with a reconstruction loss.

Benefits and results

• Much higher compression than traditional GPU formats (BCn/BC7/ASTC).
• Works well with high channel‑count materials (packed features).
• Example: Tuscan Wheels scene reduced from ~6.5 GB VRAM (BCn) to ~970 MB with NTC while maintaining comparable visual fidelity.
• Side‑by‑side comparisons show fewer compression artifacts at the same VRAM budget.

Practical benefits and availability

• Smaller disk footprint, lower download bandwidth, reduced VRAM footprint via a compute‑for‑quality tradeoff.
• Implementation: NVIDIA RTX Neural Texture Compression SDK (GitHub / QR provided in talk).

---

Case study 2 — Neural Materials

Idea

Encode material appearance (multiple layered light responses) into latent textures plus a small decoder MLP instead of storing many traditional texture maps and evaluating complex BRDF stacks.

Training architecture

• Encoder MLP (training‑only) maps material channels into a structured latent texture.
• Decoder reconstructs per‑sample BRDF/shading outputs at runtime.
• The latent bottleneck reduces memory and enforces structure.

Results and advantages

• Example: a reference material with 19 channels compressed to an 8‑channel latent representation.
• Measured render speedups (1080p, 1 spp) ranged roughly from 1.4× to 7.7× depending on the setup.
• Advantages: reduced analytic compute, lower memory bandwidth, and single‑pass decoding of multiple layers versus sequential BRDF layers.

Status

• Active research at NVIDIA and in the graphics community; promising but with limited production deployment so far.

---

Case study 3 — Neural Reconstruction for AV simulation (NeuralRec / Gaussian splatting)

Problem

Training AV policies requires vast, diverse, realistic sensor data. The sim‑to‑real gap occurs when simulated sensors don’t match real captures.

Solution: real‑to‑sim via neural reconstruction

• Represent scenes as a cloud of overlapping 3D Gaussian ellipsoid particles.
• Each particle stores: position, scale, rotation, opacity, and view‑dependent color (e.g., spherical harmonics).
• Optimize particle properties by backpropagating through a differentiable renderer to match captured images.
• Result: novel view synthesis — render from viewpoints not in the original capture.

Constraints and shortcomings

• High quality near recorded trajectories; quality degrades when extrapolating far from captured views (artifacts, missing geometry).
• Objects captured from few views (e.g., vehicles) may be incomplete.

Augmentations to handle missing data

• Neurec Fixer: a neural model (diffusion‑like / learned) that cleans up artifact‑laden renderings; can be applied at render time or offline.
• Neural Asset Harvester: detects individual objects in the reconstruction and generates completed full‑3D assets (fills unseen sides), enabling reuse and placement of complete objects anywhere in the scene.

Benefits for AV

• Photorealistic training environments, ability to create new/rare/dangerous scenarios, and a pathway to close the sim‑to‑real gap while maintaining interactive rendering performance.

Implementation note

• Gaussian splatting optimization relies on differentiable rendering — Slang auto‑diff and SlangPy are used to bridge training and deployment.

---

Tools, libraries and resources mentioned

• Slang and SlangPy — open‑source shading language with autodiff and Python bindings.
• RTX Neural Texture Compression SDK (NVIDIA / GitHub).
• RTX Neural Shaders — neural inference inside shader pipelines.
• Neurec, Neurec Fixer, Asset Harvester — research/tools (some content to be posted on Hugging Face).
• Several GTC sessions, labs, and expert pods referenced for deeper dives (links/QR codes provided in the talk).

---

Q&A highlights

• Automatic differentiation in Slang: compiler‑generated chain‑rule, internally tape‑based; similar to other AD systems though not identical.
• Fixer model foundation models: unspecified in the talk; experts at GTC can provide details.
• Fixer deployment: likely possible at render time (real‑time / interactive), but exact performance guarantees were hedged.
• Licensing: asset harvester/fixer expected to appear on Hugging Face with explicit license details when posted.

---

Main speakers / sources

• Shannon — presented neural rendering overview, real‑time constraints, Slang/SlangPy, and AV Neurec material.
• Alexey / Alexi — presented neural texture compression and neural materials.
• Additional references: NVIDIA (research & tooling), Jensen (referenced re: DLSS), Neurec / Neurec Fixer / Asset Harvester (systems described).