Shadow of HyperPose / Siggraph 2024

Summary of technological concepts, features, and analysis (HyperPose / SIGGRAPH 2024)

Purpose of the talk

• Presents HyperPose (hyper POS): a new character animation system prototype, focusing on its “shadow.”
• Positioned as early-stage and based on a “superposition” idea rather than a fully collapsed representation.
• No machine learning is used in the showcased parts; emphasis is on a human-facing framework for reasoning about motion data before optionally incorporating ML later.

Why HyperPose is called a “shadow”

• Human perception is limited: we can’t truly perceive 4D (and barely 3D).
• The “shadow” refers to representing higher-dimensional motion data through projections.
• HyperPose manages motion in a form where many possibilities exist and are only resolved via runtime mechanics/pipeline.

Core framing: animation as state evolution, not a timeline

• Reinterprets animation clips as samples of a larger entity.
• Instead of a linear sequence, HyperPose treats a pose/state as something that can evolve into other states.

---

Pipeline overview (promised progression)

The talk outlines progression through:

1. Problem-space definition
2. Solution showcase/description
3. Mechanics around HyperPose
4. High-level pipelines/workflows
5. Current progress + future goals + R&D needs

It also notes the talk is part of an Omnibus, with follow-up extensions (“Echo and Savant”) by a colleague.

---

Motivation: limitations of existing systems

Animation State Machines (ASM)

Criticized for:

• Hard splits (e.g., locomotion vs interaction) that don’t match real-world ambiguity.
• State explosion: AAA titles can have extremely large animation inventories (described as up to ~1 million frames / ~10 hours).
• Poor handling of large numbers of human actions/footwork and unreliable labeling at scale.
• “Motion tiling” artifacts: repeated pose reuse can introduce seams.

Motion Matching (MM)

Discussed as widely adopted, but argued to struggle with:

• Non-locomotion interactions and artistic agency.
• Scenarios requiring immediate responsiveness, or behaviors that become “black box” and unpredictable.
• Examples like navigating tightly crowded spaces with rich contact/interaction where no satisfactory system exists.

---

HyperPose concept: four-dimensional pose via sampling

HyperPose is described as:

• “At its core… low-dimensional sampling of high-dimensional data.”
• A representation where a finite pose can encapsulate many “shadows” (projections).

It also introduces representing poses in a shared coordinate frame by enforcing consistent placement of the root (top of the hierarchy joint).

---

Key term: PDP (Principal Dynamic Pose)

• PDP = a pose with special properties, including a velocity component.
• The system parses motion into deterministic PDPs using:
  • Similarity
  • A predecessor/successor structure (pointers to source data)
• PDPs enable building procedural state machines that generate many combinations of motion evolution at runtime.

---

Deterministic segmentation via character point cloud

To compare poses with consistent transforms:

• Uses a character point cloud (tracked body points) to derive:
  • Center of mass
  • a root coordinate frame

Claims on accuracy vs density:

• Convergence of accuracy with density: about 10 points per liter (approximately 66–60 points for an average body).

Pose comparison:

• Measures motion using displacement/velocity
• Filters noisy curves

---

Motion metrics: velocity/acceleration + “pooling”

• Computes per-time velocity
• Uses acceleration (with force/work intuition) to refine

Pooling

• Instead of noisy dense sampling:
  • Samples local minima/maxima
  • Fits into a “corridor” to preserve curve magnitude

Segmentation strategy

• Done offline
• Uses peaks/valleys of the force/acceleration-derived curve to select PDPs

Reported results

• Human-like segmentation deviation: ~1.25 frame average fit (earlier noted ~3-frame human choice deviation)

• PDP occurrences: about once per ~7 frames (large data reduction)

---

PDP storage and compression strategies

Discusses multiple approaches:

• Store PDPs only (possible fidelity loss)
• Add “scaffolding PDPs” between original PDPs to increase fidelity
• Store source frame sequences for each PDP

Overall claim: “rubber footprint compression”

• The engine can hold the full PDP structure, but the build can replace many PDPs with pointers (weighted reconstruction) to hit a target footprint (platform/LOD constraint).
• Footprint range described:
  • from about 15% of source animation data down to as little as one pose

---

Principal Dynamic Pose comparison + convergence sets

Cost and blending

• Defines a blending “cost” between PDPs:
  • Based on offset between pose candidates divided by node velocity
  • Faster motion → better blend

Multi-stage “lazy” computation

1. First compare using center of mass (eliminates most work)
2. Iterate with increasing precision
3. Finally compare peak/valley structure using point cloud comparisons

Interpretation layer (weights + local pool growth)

• Converts costs into weights
• Grows a local sample pool until coverage is complete

Convergence set

• Chooses an optimal subset of PDPs (plus weights/pointers) to approximate motion under constraints:
  • desired number of PDPs, or desired error

Analogy: “alphabet compression”

• Like reducing symbols while retaining reconstructive power.

Additional behavior:

• If two PDPs are sufficiently similar, one can be replaced with a pointer to the other; errors are still tracked.

---

Hierarchical Reduction Matrix (HRM) / nested procedural state machines

Uses precomputed reduction levels:

• Start with the full PDP set
• Merge one PDP at a time into similar counterparts
• Similarities produce weighted link graphs derived from motion data

This yields Nested Procedural State Machines:

• Low-level HRM: immediate precision
• Higher-level HRM: planning longer horizons
• Supports “zooming” between planning scales:
  • plan on higher levels, step down for near-future detail

Claimed consequence:

• Thousands of possible solutions within a short time window (e.g., ~within a second)

Debugging/preview limitation:

• No direct format for previewing nested procedural machines
• Visualization uses collapsed/unrolled 2D state machine representations

---

Showcases tied to common problems

HyperPose can:

• Prune nodes/links to reduce graph size (based on connectivity metrics, self-loop rates, etc.)

• Detect and avoid dead ends and escape loops

• Generate additional PDPs for missing coverage (exporting a Maya file for artist validation)

• Support forced-blend scenarios by selecting building blocks (PDPs) known to have high utility

• Allow pose locking by removing extra incoming/outgoing nodes around a desired target

---

Contextual runtime adaptive cost function

Runtime matching considers more than “now + target”:

• Balances secondary objectives, such as:
  • Contact with objects while moving/turning
  • Facial/head/gesture and audio matching
  • Contact transforms and time warps (not just triggering canned clips)

Implementation framing:

• Uses a contextual cost function with multiple layers of HRM and access to PDP properties stored in a struct:
  • Builds nested state machines from pointer data + HRM graph info
  • Uses metadata/tags like motion semantics (e.g., “angry sitting”, “sneeze”)

Additional capabilities mentioned:

• Space scaling (footstep scaling referenced as a prior Ubisoft concept)
• Time warping using velocity/acceleration info to avoid obvious artifacts (described as preventing “Ben Benal-like” effects)

---

Contact and interaction representation

• Contact is treated as transforms
• Potentially animated/planned, with:
  • time warps
  • metadata
• Suggests coordinate frame transformations can be adapted between character and environment objects to maintain correct contact.

---

Data requirements and motion capture for HyperPose

• Emphasizes that abundant motion data is critical.
• Mentions developing a new motion capture scheme:
  • “Loop” (to be presented later alongside “Echo” and “Savant”).

Data growth analysis

• PDP count grows roughly linearly with keyframes
• Convergence set size grows logarithmically with data (implying a plateau)
• Redundancy increases with more data, with diminishing returns

Scalability estimates

• “Complete human HyperPose” estimated around ~1500 PDPs (some uncertainty)
• Stores ~minutes of compressed data for ~10 hours of source motion

---

Claimed runtime capabilities

Compared to motion matching, HyperPose aims to guarantee wider reachability:

• Instead of “blind spots” in state space, it accumulates hits per unit of space to show reach potential.

Reported examples:

• Tight navigation plus world interactions
• Multiple goals simultaneously
• Already running on mobile on ~50 characters

---

Planned future work and risks

Future work

• Improve user-facing tooling for constructing/visualizing state machines while preserving human agency.
• Resolve storage/asset pipeline questions (e.g., scaffolding vs directional initialization vs alternatives).
• Push compression further to bring mobile closer to high-end parity.
• Audio extensions:
  • match full-body motion with spatial trajectories and amplitude
  • acknowledges speaking-while-locomoting complexity
• Guidance/navigation:
  • critiques rigid traditional AAR/navmeshes
  • interested in kinematic approaches with bidirectional exploration

Risk: “Arcade Physics Paradox”

• Player requests (e.g., double jumping onto impossible actions, violating momentum rules) conflict with realism.
• Suggested mitigation via HyperPose extensions like time warping and Echo.

---

Main speakers / sources

• Alex Bnak — Director of Central Animation Technology team (CAT), associated with Activision Blizzard King / Xbox Studios / Ubisoft (as referenced).
• MDI Farala — mentioned as colleague for follow-up talks (“Omnibus extensions… Echo and Savant”).

Mentioned historical/technical influences and names

• Michael Butner (motion fields / motion matching origin story)
• Simon Clavier
• Michael(?) JC Delano, Jonathan Cooper (referenced among motion matching contributors)
• David Boll (Epic Games) for an initialization/storage approach
• Disney’s 12 Principles of Animation
• Rako (quoted in ASM discussion)
• S. Hawking and George Carlin (used for humorous framing of presentation style)
• Kinematics (referenced as a method for navigation)