THIS is the Biggest Thing Since CGI

Overview

Gaussian splats are an explicit, GPU-friendly 3D representation made of many fuzzy 3D blobs (“Gaussians” or “splats”) that store position, orientation, shape, opacity and view-dependent color. They enable photoreal, real-time 3D reconstructions and holograms.

This document summarizes key technical concepts, workflows, product notes, strengths/limitations, and practical tips for creating and using Gaussian splats (including 4D/temporal variants).

What Gaussian splats are

• An explicit radiance-field / hologram representation composed of many overlapping 3D Gaussians (splats).
• Each splat encodes:
  • Position, orientation, shape, opacity.
  • View-dependent color encoded compactly with spherical harmonics (small coefficient set per splat).
• Contrast with NeRFs:
  • NeRFs are implicit neural functions (slow, hard to edit).
  • Gaussian splats are explicit point-like objects and are GPU-friendly, enabling real-time rendering in browsers, engines, and VR headsets.

How they work (high level)

1. Capture many overlapping images (or 360 footage / multi-camera rigs).
2. Run structure-from-motion (SfM) to recover camera poses and a sparse point cloud.
3. Grow Gaussians on the point cloud and optimize them to reproduce captured views — they “splatter” together to form smooth photoreal imagery.
4. For 4D capture, Gaussians include velocity and lifetime metadata so points interpolate/hand-off across time instead of being independent per frame.

• Result: continuous motion, arbitrary playback/frame rates (including ultra-high FPS), and much smaller file sizes compared to naive per-frame representations.

Capture → Tracking → Training (pipeline & practical tips)

• Capture
  • Treat footage as a dataset, not a single video.
  • Use crisp images: high shutter speed (e.g., 1/500–1/1000), lock white balance, ISO, focus.
  • Avoid motion blur and changing lighting.
  • For static scenes: many photos from multiple angles. For spaces: 360 cameras or multi-camera rigs help.
• Tracking
  • Run SfM to get camera poses and a sparse point cloud.
  • Tools vary for equirectangular/360 footage — Metashape supports spherical maps; otherwise convert to cube faces or use scripts to extract trackable images.
• Training
  • Many tools automate training once poses/dataset are prepared.
  • Output is the Gaussian splat representation ready for viewing or rendering.

Software, services and common workflows

• Capture / Tracking / Training tools:
  • PosedShot (paid, all-in-one)
  • Lick Field Studio / Light Field Studio (open source, requires external tracking)
  • RealityScan, Colmap, Curie Engine, Luma, Metashape (360 tracking)
  • Antigravity Studio (drone footage framing/export)
• Sharing / viewing:
  • Super Splat (examples / gallery)
• Rendering / production:
  • Unreal Engine (real-time playback, LED-wall integration)
  • Octane Render (path-traced relighting, effects)
  • Web browser viewers and VR headsets (e.g., Meta Quest 3)
• 4D / volumetric capture providers:
  • 4D V.AI (aka 4D Views?) — stage capture with dozens of synchronized cameras and cloud processing for 4D Gaussians

Product review / guide highlights (from the video)

• Capture best-practices: shutter speed, locked settings, dataset mindset, 360 vs handheld, and using photo modes or renders to create synthetic datasets.
• Software guidance: trade-offs between all-in-one paid solutions (easier) vs. free/open-source workflows (manual tracking required).
• Antigravity A1 drone review:
  • 360 drone optimized for capture with features like SkyGenie (auto camera moves), SkyPath (keyframed flight paths), Deep Track (subject following), and equirectangular export.
  • Practical benefit: rapid multi-angle 360 capture for splats.
• Demonstrations / techniques:
  • Convert high-detail renders and game photo-mode outputs into splats to make them interactive real-time 3D assets.
  • Relighting, path-tracing, adding 3D objects, refraction (when path-traced), depth-of-field, and creative VFX (bending worlds, miniature/tilt-shift shots).

4D Gaussian splats — technical & operational highlights

• Temporal continuity:
  • Gaussians carry velocity and lifetime so points interpolate across time; this enables seamless slow motion at arbitrary FPS and avoids frame-to-frame artifacts.
• Compression:
  • Continuous temporal representation allows compression to ~30–60 Mbps for 4D splats, reportedly ~100× smaller than raw 2D capture data — streamable to phones and headsets.
• Use-cases:
  • Volumetric actor capture with clean alpha mattes (green-screen replacement)
  • LED-stage integration, VR/AR experiences
  • Historical preservation, mapping/real-estate (Google Maps / Zillow)
  • Film/VFX and interactive installations

Strengths and limitations

Strengths

• Photoreal, real-time rendering with good view-dependent effects (reflections, sheen).
• Preserves high-frequency details (hair, leaves, insect wings).
• Editable in rendering engines; can be path-traced for correct lighting/refraction in tools like Octane.
• Continuous 4D representation enables extreme slow-motion and flexible playback rates.

Limitations

• Anything not captured is missing (occluded or unseen areas must be captured explicitly).
• Not full physics-based refraction out-of-the-box — translucency/reflection approximated unless re-rendered in a path-tracer.
• High-quality 4D capture currently requires many cameras and significant compute to train, though these friction points are improving.

Practical examples demonstrated

• Super Splat gallery scans: garage, chocolate/foil, curtain, TV reflections (mirror-world effect), dog hair depth, bee wing translucency.
• Converting renders and game photo-mode captures into splats for interactive viewing.
• Studio 360 capture with Antigravity A1 drone.
• Full 4D capture on a volumetric stage (e.g., 96 cameras at 60 FPS) processed into a 4D Gaussian splat for live-interview holograms and LED-wall use.
• Relighting and path-traced edits: adding sunlight, fog, firelight, crystal-ball refraction, and other VFX in Octane/Unreal.

Future outlook

• Rapid adoption expected in VR/AR, mapping, real-estate, historical preservation, and live volumetric capture.
• Key bottlenecks (camera count, capture complexity, training compute) are actively being reduced and may diminish in 1–2 years.
• Gaussian splats are positioned as a likely mainstream holographic medium.

Tutorial / Checklist (concise)

1. Capture
   • High shutter speed; lock white balance, ISO, focus.
   • Lots of overlapping viewpoints (or 360/drone); avoid motion blur and changing light.
2. Tracking
   • Run SfM (Metashape, Colmap, etc.) to get camera poses and a sparse point cloud.
3. Training
   • Use an all-in-one solution (PosedShot) or combine tracking + a light-field trainer to generate splats.
   • Test in a viewer, then import to Unreal/Octane for editing or deployment.
4. Tips
   • Use renders/game photo-modes to create splats of CG objects.
   • Reprocess old photogrammetry datasets to upgrade to splats.
   • If using 360 where you appear in capture, mask yourself out.

Main speakers / sources

• Video host / narrator (YouTuber demonstrating workflows and tools)
• Jiaming — CEO of 4D V.AI / representative of 4D Views (interviewee on 4D Gaussian splats)
• Mentioned tools and companies: PosedShot, Lick Field (Light Field) Studio, RealityScan, Colmap, Curie Engine, Luma, Metashape, Super Splat, Antigravity (A1 drone & Antigravity Studio), Octane Render, Unreal Engine, 4D V.AI / 4D Views.