Tools for developing for HoloLens 2 - Unite Copenhagen

Summary of the Talk: Tools for Developing for HoloLens 2 with Unity (Unite Copenhagen)

Speaker Background / Motivation

• Nick (from a game development background) explains how Unity enabled tool development as a career through the Asset Store, creating an ecosystem where tools can be shared, maintained, and improved.
• After joining Microsoft ~1.5 years ago, he works on HoloLens 2 and describes Microsoft-wide efforts to make mixed reality (MR) development easier, largely by releasing open-source tools on GitHub (encouraging issues and pull requests).

---

HoloLens 2 Hardware Capabilities (Why MR Dev Is Different)

The talk highlights key sensor/display features MR developers must design for:

• Eye tracking: high accuracy, effective “at arm’s length.”
• Embedded Kinect sensor: supports room scanning and helps maintain an in-memory representation of the environment.
• Very strong hand tracking: continuously improving over time.
• Stereoscopic display:
  • 60 Hz
  • readable text resolution
  • comfortable viewing experience
  • large field of view
• Comfort & battery: lasts about 3 hours on a single charge (as stated).

---

Flagship Developer Tool: Mixed Reality Toolkit (MRTK) for Unity

Nick’s main focus is Microsoft’s Mixed Reality Toolkit (MRTK) as a Unity extension.

MRTK provides:

• Interaction paradigms
• “Smart distillation” of complex hardware constraints into developer-friendly APIs (including handling edge cases for camera/hand/eye inputs)

It also includes major subsystems:

1) Mixed Reality Standard Shader

• Designed as a direct replacement for Unity’s standard shader to address mobile GPU framerate issues (e.g., Snapdragon 850 context).
• The talk claims a significant performance boost when swapped in.
• Additional rendering features include:
  • Stencil support
  • Distance/disc clipping (for “peering inside” meshes)
• Mentioned as a useful reference implementation for shader families, with code available.

2) Robust Input Abstraction (Cross-Platform Foundation)

• Converts diverse input methods into a consistent action-based input model.
• Emphasizes cross-headset portability—not just HoloLens 2 (“works on pretty much any headset”).
• Notes ongoing ecosystem expansion, including support for phone AR (“coming”).

3) Standard UI Components

• UI components designed to “look at home” in the HoloLens shell environment.
• Based on internal experience to provide polished interaction and placement behaviors, reducing the need to reinvent common UI patterns.

4) Eye-Focused Interaction Examples

Includes MR examples and tooling such as:

• Gaze heatmaps (where people look)
• Eye-based text interactions (e.g., auto-scrolling)
• Zooming relative to gaze (no need to manually select a zoom point)
• Gaze + tap selection as a fast selection paradigm
• Notes gaze can provide context for commands (e.g., “make that bigger” because the user is already looking at it)

5) Learning Modules / Tutorials

• Newly released MR Learning Modules for MRTK v2 on GitHub.
• Encourages community participation:
  • file issues
  • watch triage
  • contribute via forks

---

Other Major Tools & Technologies Mentioned

Mixed Reality Spectator View (Revised)

• A rewritten spectator/casting tool for showing what others are seeing.
• Example workflow:
  • Track interactions using a phone with Azure Spatial Anchors
  • Localize and stream captured hologram interactions to another display
• Described as easier to integrate because it:
  • “scrapes the scene”
  • syncs elements
  • shares over the network
• Still requires embedding inside the application, but aims to reduce complexity.

Built-in QR Tracking (Low Overhead)

• Uses tracking cameras (not the main camera) at about 2 Hz (“negligible cost”).
• Motivation:
  • avoid CPU overhead
  • avoid user discomfort from turning on/using the main camera

Azure Spatial Anchors (Cloud-Based Environment Sharing)

Presented as a cross-platform anchor system:

• Scan environment → build anchor → upload to cloud → share/localize across devices.
• Emphasized as scalable and suitable for live services.

Examples:

• Minecraft Earth uses geospatial anchoring (“anywhere on the planet”).
• Supports relationships between anchors enabling wayfinding.

Havok Physics Plugin for Unity (Better Hand-Object Interaction)

• Adds Havok physics to Unity instead of using Unity’s default physics.
• Key concept: penetration recovery—when virtual objects overlap, the simulation smoothly resolves overlap (avoiding unrealistic forces).
• Positioned as beneficial for high-quality, physically believable hand interactions (including cases like tracked “God hand” issues).

Microsoft Game Stack (Platform Ecosystem)

Described as “games adjacent technology,” including:

• Project Acoustics: “audio light-mapping”
  • Includes a Unity plugin
  • Builds a representation of a room to improve spatial audio realism
• Sure / PlayFab: mentioned as tools for accelerating multi-user workflows and multiplayer/multisession experiences

Mixed Reality WebRTC

• Built for streaming video/audio between devices (e.g., laptop ↔ HoloLens).
• Framed as previously difficult to implement, but now available as a ready-to-use Unity-oriented solution.
• A latency Q&A notes latency depends on the underlying tech; it also mentions Miracast as a possible alternative with faster latency, without providing exact numbers.

Mixed Reality Lighting Tools (Speaker’s Work)

Focuses on lighting for additive displays (HoloLens optics).

Approach:

• Uses camera + orientation to generate/update cube maps
• Estimates primary light direction over time
• Uses it to drive shadows and immersion
• Also described as helping compensate for real-world lighting differences (e.g., bright vs dark sides)

Mixed Reality Extension SDK (AltspaceVR Heritage)

• Supports server-hosted multi-user experiences with session spawning and shared interactions.
• Includes a Unity SDK path to use extensions inside your own apps.

Microsoft Maquette (VR Prototyping Tool)

• A VR modeling/prototyping experience:
  • supports rapid creation at scale
  • supports creation from multiple viewpoints
• Includes a Unity plugin so prototypes can be imported while preserving hierarchy for further scripting.

---

Voice / Language Tools

Microsoft Speech Services

• Provides:
  • speech-to-text
  • text-to-speech
  • translation
  • natural language understanding
• Example: two users speaking different languages can translate over the network.

LUIS (“LUIS” Mention)

• An intent-based conversational/voice interface system.
• Supports training/refinement over time using user interactions and review/refinement sessions.

---

Development Strategy for Upcoming Headsets

• Suggests starting with VR headsets first, then porting to HoloLens 2 once it’s available (“plop things onto there” approach).
• Recommends keeping MR development flexible while targeting HoloLens 2 as the eventual endpoint.

---

Key Links Mentioned

• The talk ends with a reference to a public list of links to the tools discussed.
• Contact options mentioned:
  • Twitter
  • email (for development questions)

---

Main Speakers / Sources (As Mentioned)

• Nick (main speaker; worked on HoloLens 2 tools; from game development)
• Cameron Mica (credited for the Mixed Reality Standard Shader work)
• Jerome (mentioned in the context of Mixed Reality WebRTC)
• Microsoft (overall organization providing tools and open-source distribution)
• GitHub / Azure (platforms referenced for open source, anchors, and services)