How The Best Engineers Solve Problems Most Give Up On (Niantic Spatial CTO)

Technological concepts & product/engineering focus (Niantic Spatial CTO)

4D world model as the long-term goal

• Build a “four dimensional view of the world” by combining:
  • Base maps (e.g., Google Earth-like snapshots)
  • Historical and live imagery
• The system should answer questions like:
  • “What did this place look like on Jan 3rd / Aug 15th?”
  • Using photo timestamps and positions
• The intent is temporal change understanding, not just visualization.

Localization improvements → research-to-production

• Create a new world model for visual localization:
  • Matching a single photo to a location in a spatial map
• Reported performance:
  • ~32–100% better recall/precision for localization from one photo
• Emphasis on delivery:
  • Turning a researcher-created idea (started Feb, shipped Aug) into a production service
  • Target: live to customers by early next week
• Key message: production readiness matters more than “paper/check-the-box” research.

Ground photo + satellite photo matching

• Match a ground-level photo to a high-resolution satellite image (example mentioned: Venter imagery)
• Goal: determine exact relative position
• Purpose: support the 4D timeline idea by tying multiple sensing modalities to a consistent geographic frame.

Precision claims for spatial measurement

• Accuracy varies by map/location, commonly cited as:
  • ~half a meter / half a degree
• In some cases, accuracy can reach centimeters (example context referenced: specific objects like “ingredients”).

How the system gets its “richness” (data pipelines)

• Leverages existing sources:
  • Legacy imagery
  • Photogrammetry reconstructions (e.g., historical aerial imagery used for 3D building reconstruction)
• Key technical change:
  • Machine learning depth models (example named FMS depth model)
  • Estimate depth from one or multiple photos to enable denser and previously impossible reconstructions
• Enables “go back in time” reconstruction:
  • Reconstruct from older photos/video, including decades-old imagery.

Organization tools for enterprises

For customers with large unorganized collections of photos/3D scans, Niantic provides tools to:

• Organize spatial content into a usable database-like structure (not a pile of images)
• Lock photos from different years to the same physical place/orientation
  • Enabling change-over-time
• Target workflows:
  • Compliance/repair
  • Investigations (construction, factories, etc.)

Change detection + semantic queries (“make it a report, not a slider”)

Instead of manual eyeballing with day-one/day-two sliders, the goal is to:

• Run semantic segmentation
• Perform change detection
• Produce human-readable outputs such as:
  • “cement mixer moved from zone A to Z during day two”
  • Flag safety violations (OSHA-like requirements)

Some queries are described as:

• Easy (e.g., “cement mixer moved”)
• Harder because they require deeper scene understanding and policy-level interpretation, e.g.:
  • “site up to OSHA cleanliness requirements”

---

Robotics & simulation integration

Gaussian Splatting for accurate robot simulation assets

A delivered capability turns real rooms into assets usable by robotics simulators:

• Output includes:
  • A Gaussian splats representation packaged into a USDZ file
  • A collision mesh (for feet/wheels interactions)
  • Designed to load directly into NVIDIA Isaac simulator
• Rationale:
  • Reduce the sim-to-real gap by making simulated environments visually correct enough
  • Improve robot training when deployed
• What Gaussian splats are (vs. traditional assets):
  • Not triangle meshes or traditional texture maps
  • Uses Gaussian primitives with variable size
  • Supports view-dependent color, improving realism for:
    • glass, water, reflective materials
  • Fast enough for high frame rates on GPUs and within simulators for rapid training

---

Scale & remaining unsolved problems

Hard part is scale (not just a single room)

• Building a detailed 4D model for a room is manageable
• Scaling to hundreds of km² (example: Rancho Cordova, ~50 km²) is harder
• Challenges include:
  • Pulling in and locating where photos exist
  • Organizing and modeling imagery at large geographic scale
  • Turning it into semantic understanding of what changed between dates

From pictures to semantic/causal analytics

Next steps beyond accurate models:

• Segmentation + change detection at scale
• Enable AI to answer questions about changes and possible causes
  • Example idea: analyze semantic change events (e.g., speeding ticket rates) and infer explanations via an AI model plus structured extracted changes.

AI query reliability + financial viability

Two constraints emphasized:

• Speed: answers must be fast enough to “keep up” with AI-driven queries
• Cost: answers must be produced financially viable

---

AI engineering principles / CTO guidance

“Token maxing is bullshit”

• Don’t use AI “for the sake of AI”
• Not all problems will be solvable soon, but some subsets will be

Design the problem space for AI

• Frame tasks so they are:
  • Self-checkable with measurable ground truth
  • Backed by evaluation datasets
• Include test/validation mechanisms so the AI can:
  • Verify correctness
  • Avoid hallucinated success

Model-usage intuition matters

• Don’t blindly stack many model variants
• Different model versions have different strengths/weaknesses
• Maintain awareness of model behavior changes across versions (examples mentioned: Claude, Gemini, and references to 4.x / 5.0-type versions)

---

Main speakers / sources

• Brian McClendon — described as an engineer behind Google Earth, previously ran Google Maps, and now CTO at Niantic Spatial.