Summary of "THIS is the Biggest Thing Since CGI"

Overview

Gaussian splats are an explicit, GPU-friendly 3D representation made of many fuzzy 3D blobs (“Gaussians” or “splats”) that store position, orientation, shape, opacity and view-dependent color. They enable photoreal, real-time 3D reconstructions and holograms.

This document summarizes key technical concepts, workflows, product notes, strengths/limitations, and practical tips for creating and using Gaussian splats (including 4D/temporal variants).

What Gaussian splats are

An explicit radiance-field / hologram representation composed of many overlapping 3D Gaussians (splats).
Each splat encodes:
- Position, orientation, shape, opacity.
- View-dependent color encoded compactly with spherical harmonics (small coefficient set per splat).
Contrast with NeRFs:
- NeRFs are implicit neural functions (slow, hard to edit).
- Gaussian splats are explicit point-like objects and are GPU-friendly, enabling real-time rendering in browsers, engines, and VR headsets.

How they work (high level)

Capture many overlapping images (or 360 footage / multi-camera rigs).
Run structure-from-motion (SfM) to recover camera poses and a sparse point cloud.
Grow Gaussians on the point cloud and optimize them to reproduce captured views — they “splatter” together to form smooth photoreal imagery.
For 4D capture, Gaussians include velocity and lifetime metadata so points interpolate/hand-off across time instead of being independent per frame.

Result: continuous motion, arbitrary playback/frame rates (including ultra-high FPS), and much smaller file sizes compared to naive per-frame representations.

Capture → Tracking → Training (pipeline & practical tips)

Capture
- Treat footage as a dataset, not a single video.
- Use crisp images: high shutter speed (e.g., 1/500–1/1000), lock white balance, ISO, focus.
- Avoid motion blur and changing lighting.
- For static scenes: many photos from multiple angles. For spaces: 360 cameras or multi-camera rigs help.
Tracking
- Run SfM to get camera poses and a sparse point cloud.
- Tools vary for equirectangular/360 footage — Metashape supports spherical maps; otherwise convert to cube faces or use scripts to extract trackable images.
Training
- Many tools automate training once poses/dataset are prepared.
- Output is the Gaussian splat representation ready for viewing or rendering.

Software, services and common workflows

Capture / Tracking / Training tools:
- PosedShot (paid, all-in-one)
- Lick Field Studio / Light Field Studio (open source, requires external tracking)
- RealityScan, Colmap, Curie Engine, Luma, Metashape (360 tracking)
- Antigravity Studio (drone footage framing/export)
Sharing / viewing:
- Super Splat (examples / gallery)
Rendering / production:
- Unreal Engine (real-time playback, LED-wall integration)
- Octane Render (path-traced relighting, effects)
- Web browser viewers and VR headsets (e.g., Meta Quest 3)
4D / volumetric capture providers:
- 4D V.AI (aka 4D Views?) — stage capture with dozens of synchronized cameras and cloud processing for 4D Gaussians

Product review / guide highlights (from the video)

Capture best-practices: shutter speed, locked settings, dataset mindset, 360 vs handheld, and using photo modes or renders to create synthetic datasets.
Software guidance: trade-offs between all-in-one paid solutions (easier) vs. free/open-source workflows (manual tracking required).
Antigravity A1 drone review:
- 360 drone optimized for capture with features like SkyGenie (auto camera moves), SkyPath (keyframed flight paths), Deep Track (subject following), and equirectangular export.
- Practical benefit: rapid multi-angle 360 capture for splats.
Demonstrations / techniques:
- Convert high-detail renders and game photo-mode outputs into splats to make them interactive real-time 3D assets.
- Relighting, path-tracing, adding 3D objects, refraction (when path-traced), depth-of-field, and creative VFX (bending worlds, miniature/tilt-shift shots).

4D Gaussian splats — technical & operational highlights

Temporal continuity:
- Gaussians carry velocity and lifetime so points interpolate across time; this enables seamless slow motion at arbitrary FPS and avoids frame-to-frame artifacts.
Compression:
- Continuous temporal representation allows compression to ~30–60 Mbps for 4D splats, reportedly ~100× smaller than raw 2D capture data — streamable to phones and headsets.
Use-cases:
- Volumetric actor capture with clean alpha mattes (green-screen replacement)
- LED-stage integration, VR/AR experiences
- Historical preservation, mapping/real-estate (Google Maps / Zillow)
- Film/VFX and interactive installations

Strengths and limitations

Strengths

Photoreal, real-time rendering with good view-dependent effects (reflections, sheen).
Preserves high-frequency details (hair, leaves, insect wings).
Editable in rendering engines; can be path-traced for correct lighting/refraction in tools like Octane.
Continuous 4D representation enables extreme slow-motion and flexible playback rates.

Limitations

Anything not captured is missing (occluded or unseen areas must be captured explicitly).
Not full physics-based refraction out-of-the-box — translucency/reflection approximated unless re-rendered in a path-tracer.
High-quality 4D capture currently requires many cameras and significant compute to train, though these friction points are improving.

Practical examples demonstrated

Super Splat gallery scans: garage, chocolate/foil, curtain, TV reflections (mirror-world effect), dog hair depth, bee wing translucency.
Converting renders and game photo-mode captures into splats for interactive viewing.
Studio 360 capture with Antigravity A1 drone.
Full 4D capture on a volumetric stage (e.g., 96 cameras at 60 FPS) processed into a 4D Gaussian splat for live-interview holograms and LED-wall use.
Relighting and path-traced edits: adding sunlight, fog, firelight, crystal-ball refraction, and other VFX in Octane/Unreal.

Future outlook

Rapid adoption expected in VR/AR, mapping, real-estate, historical preservation, and live volumetric capture.
Key bottlenecks (camera count, capture complexity, training compute) are actively being reduced and may diminish in 1–2 years.
Gaussian splats are positioned as a likely mainstream holographic medium.

Tutorial / Checklist (concise)

Capture
- High shutter speed; lock white balance, ISO, focus.
- Lots of overlapping viewpoints (or 360/drone); avoid motion blur and changing light.
Tracking
- Run SfM (Metashape, Colmap, etc.) to get camera poses and a sparse point cloud.
Training
- Use an all-in-one solution (PosedShot) or combine tracking + a light-field trainer to generate splats.
- Test in a viewer, then import to Unreal/Octane for editing or deployment.
Tips
- Use renders/game photo-modes to create splats of CG objects.
- Reprocess old photogrammetry datasets to upgrade to splats.
- If using 360 where you appear in capture, mask yourself out.

Main speakers / sources

Video host / narrator (YouTuber demonstrating workflows and tools)
Jiaming — CEO of 4D V.AI / representative of 4D Views (interviewee on 4D Gaussian splats)
Mentioned tools and companies: PosedShot, Lick Field (Light Field) Studio, RealityScan, Colmap, Curie Engine, Luma, Metashape, Super Splat, Antigravity (A1 drone & Antigravity Studio), Octane Render, Unreal Engine, 4D V.AI / 4D Views.