3D Gaussian Splatting from Hollywood Films!

Summary of Technological Concepts / Product Features / Analysis

What 3D Gaussian Splatting Enables (Core Promise)

The video demonstrates converting real movie footage—specifically shots where the camera moves—into 3D reconstructions using 3D Gaussian Splatting. The goal is to bring the reconstruction into Unreal Engine 5, then:

• change camera angles,
• explore the scene,
• and reuse the result for new creative output.

High-Level Training Pipeline

1. Select suitable movie shots
   • Camera motion should be moderate (to avoid motion blur).
   • Prefer shots where only the camera moves, with minimal subject motion (moving people/objects introduce artifacts).
2. Convert footage to an image sequence
   • Extract frames into an image sequence.
3. Train the 3D Gaussian Splat model
   • Start from images to form point clouds (via the algorithm).
   • Convert point clouds into Gaussian “splats” that encode:
     • color
     • transparency
     • varying sizes
   • Optimize repeatedly until the reconstruction is photo-realistic and renders efficiently.
4. Use the reconstructed splats in Unreal Engine 5

Shot Selection Criteria / Troubleshooting Insights (From Experiments)

• Shots with no moving elements besides the camera generally produce the best reconstructions.
• Motion blur and extra moving objects cause:
  • ghosting,
  • crisscross artifacts,
  • duplicated/shifted elements,
  • and sometimes fully unusable results.
• Even with degraded inputs (e.g., YouTube-compressed video exported multiple times), results can still be surprisingly good—though artifacts increase.

Demonstrated Workflow Inside Unreal Engine 5

• Re-lighting the splats
  • Remove an existing directional light.
  • Add a single point light, adjusted warmer/cinematically.
• Scene augmentation
  • Add new objects (example: a chair).
  • Create/modify 3D content from a logo.
• Logo-to-3D tutorial mention
  • References a workflow using Inkscape + Blender to turn a logo into 3D.
  • Then import into Unreal Engine 5 and place it in the scene.

Creative Recreation Examples Mentioned

• Harry Potter
  • Create a “game-like level” view.
  • Add 3D text using the Harry Potter font.
  • Adjust Niagara particles (a visual styling tweak by changing particle sprite size).
• The Shining
  • Change camera movement (e.g., slide from closeup back).
  • Use the original Shining poster font.
  • Add fog to stylize the render.

Key Shot Results (Reviews / Reconstruction Quality Analysis)

1. The Shining (Best / very strong)

   • Input: camera orbit around the hotel; crisp features (mountains/trees/cars).
   • Output: reconstruction supports going in/out, rotating the camera, and exploring beyond the original framing.
   • Limitation: you cannot see areas not captured in the original shot (e.g., “can’t go up” if that side wasn’t recorded).

2. Glorious Bastards (Decent but needs careful testing)

   • Input behavior: slow, subtle camera orbit; minimal character movement to reduce artifacts.
   • Output: “quite decent,” though quality drops when faces/items shift across angles.

3. Harry Potter (Quite decent; noisy inputs still workable)

   • Input issues: images criticized as noisy/grainy and low-quality due to compressed YouTube footage.
   • Output: despite artifacts, reconstructs major structures (e.g., the castle/Hogwarts) and shows an apparent “denoising pass” effect.

4. The Dark Knight (Mixed / some angles good, others bad)

   • Suspected cause: heavy zoom-in reduces usable coverage.
   • Output: some angles look phenomenal (materials/details visible), while other parts look messed up (including crisscross-like artifacts).

5. Prince of Persia (Generally good; motion blur remains a problem)

   • Input issues: fast motion, motion blur, plus obstacles (pillars).
   • Output: strong depth and geometry; a notable artifact appears on the top of the head.

6. The Matrix (Mostly unusable due to moving elements)

   • Attempt 1: ghosting/double images on Trinity’s hands/wrist (likely from motion blur / temporal effects).
   • Attempt 2: bullets and other moving elements; result contains heavy artifacts.
   • Conclusion: Matrix reconstructions are not usable, while those without moving elements are better.

Main Takeaways

• Shot selection matters more than the model itself. Minimal scene motion + moderate camera speed → far better Gaussian splat reconstructions.
• Artifacts correlate with motion: motion blur and moving objects lead to ghosting, crisscrossing, and duplicated silhouettes.
• Once reconstructed, Gaussian splats are flexible in Unreal Engine: you can re-light, add objects, and produce new creative shots.

Main Speakers / Sources (As Referenced)

• Primary speaker(s): a presenter appears throughout the video and mentions “Farad” as an additional key person in the process.
• External source referenced: “Raven… (Far) Farad” is described as an amazing YouTuber who made a tutorial on converting logos to 3D using Inkscape and Blender (exact channel name not clearly captured).