Video summary

Next-Gen AI Architecture Through Co-Packaged Optics

Main summary

Key takeaways

Technology

Tech focus: Co-Packaged Optics (CPO) for next-gen AI systems

The video discusses how co-packaged optics (CPO) can evolve AI architectures by moving optical interconnects closer to compute (switches/GPUs/TPUs). This is framed as a response to the growing I/O bandwidth wall as AI demand increases rapidly.

Speakers present CPO as a multi-generational technology—not just a near-term product—needed to justify major infrastructure investment (CAPEX, supply chain readiness, and testing/packaging readiness).

Key points: Proposed evolution path for optical interconnects

Speakers outline a phased progression (not instantaneous):

  1. CPO embedded on the switch ASIC / switch side

    • Considered an earlier, “easier” insertion point due to lower volumes and a strong need to extend beyond copper reach without repeaters/equalization.

  2. Optical interconnects moving into compute packages

    • Next step: integrate optics into compute engines (GPUs/accelerators), targeting scale-up architectures.

  3. Optical links supporting extended memory

    • Ultimately, CXL-based extended/disaggregated memory over optics is discussed as a “harder” step, requiring both low latency and full platform readiness.

Scale-out vs. Scale-up: why CPO is harder in scale-up

A core analysis compares CPO adoption across two domains:

Scale-out (mature proving ground)

  • Optical is already required due to the bandwidth-distance product.
  • This environment is seen as commoditized/proven.
  • It acts as a “proving ground/pipe cleaner” for technologies later deployed at higher density and complexity.

Scale-up (next-order difficulty)

Scale-up to rack-level systems is described as harder across three pillars:

  1. Technology differences

    • Scale-up needs many more lanes than scale-out’s typical lane counts.
    • Requires custom modulator and coupling technologies plus tighter co-packaging.

  2. Advanced packaging flow + testing

    • In scale-out, module makers can integrate optics differently across mechanical/thermal/electrical approaches.
    • In scale-up, packaging/test integration must work seamlessly with switch/CPU integration and rack integration, making it less forgiving.

  3. Supply chain & ecosystem

    • Scale-up volumes require a mature, sustainable supply chain to support massive deployment.

Speakers estimate scale-up demands and system stakes rise by ~10–100× (bandwidth/volume) versus scale-out.

Why optics (and CPO) now: bandwidth cadence vs packaging cadence

From the perspective of accelerator/compute-engine builders:

  • Accelerator generations arrive roughly every 1.5–2 years.
  • Copper/packaging evolution (more layers, higher signaling rates) can’t keep up due to signal/power integrity challenges at higher layer counts and densities.
  • CPO is positioned as a shift to a more efficient “channel” (optics with lower losses/parasitics), even though it introduces new challenges.
  • Key benefit: CPO supports system-defined, optimized solutions in controlled deployment environments (scale-up “closed” systems) rather than open, multi-vendor interoperability.

Multigenerational investment: enabling long-lived platform CAPEX

Multiple speakers emphasize CPO’s potential to support many generations (similar to how copper once did). Key rationale:

  • CPO provides more degrees of freedom, such as:
    • wavelength count
    • modulation rate
    • connector density/ports
    • polarization states
  • Telecom-style component approaches (e.g., micro-ring concepts) and advanced chip stacking are seen as drivers of scalable density.
  • Because supply chain, connectors, testing, and rack integration are expensive, the investment is only justified if the optics platform can last for ~2 decades of potential use.

Interoperability vs “multiple flavors”

The panel discusses standardization challenges:

  • UCI / bus-style interoperability is described as a “wild west,” with variants optimized differently for bandwidth/power/packaging.
  • Speakers argue hyperscalers likely won’t fully standardize early, because they can tune vendor solutions to their needs and optimize for speed/cost.
  • Still, some commonality is expected at certain abstraction levels, such as:
    • shared form factor
    • shared “certified” manufacturing/test ecosystems (analogous to OSAT/certified supply chain)
    • limited tolerance for variation (small deviations acceptable; major process/form-factor differences not)

Analogy: while CPU/NUMA designs can be idiosyncratic on motherboards, CPO introduces light pipes and physical optical compatibility constraints, forcing more alignment.

Who adopts CPO first?

Adoption is expected to come from the actor that has:

  • the need (domain sizes, egress density),
  • the ability to pay, and
  • the practical deployment capability.

No explicit “confirmed winner” is named, but the discussion suggests hyperscalers/major cloud builders will lead, alongside efforts to build an “open ecosystem.”

Extended memory (CXL over optics): harder step after optical scale-up

A dedicated segment covers CXL-based disaggregated/extended memory:

  • It’s considered harder than simply using optical links between compute and switch.
  • The challenge is driven by tighter latency constraints and additional memory-protocol complexity.

Staged roadmap proposed:

  1. First integrate optics into GPU/accelerator scale-up paths and validate performance.
  2. Then add CXL-style extended memory—potentially using DWDM-like approaches (vs CWDM) for multi-wavelength transport.

A timeframe estimate is mentioned: roughly a half-generation to a generation after initial optical scale-up—about ~1 to 1.5 years offset between early CWDM-like deployments and later DWDM-like fast-followers.

Speakers connect this to memory market cadence (e.g., movement from HBM generation to newer HBM densities) and model evolution (training vs inference inflection).

Reliability, failure modes, and graceful degradation

Optical-specific risks are discussed:

  • Optical introduces new failure points, including fiber attachment/connectors and opto-electronic components like lasers.

Speakers suggest considering failure modes in two phases:

  • Startup / rack construction failures
    • fiber cleanliness/connectivity and assembly/test issues
  • Reliability over time
    • component wear-out and interface degradation

Compared to copper, optical requires:

  • more stringent qualification
  • stronger software/telemetry-driven detection and remediation

Consensus: systems must expose telemetry at the lane/link/laser level, enabling software-configurable detection and action. Some mention also includes modular/pluggable laser approaches to reduce heat and support replaceability.

Packaging/manufacturing cost: “value chain ownership” and design-for-manufacturing

A Q&A addresses why packaging innovations are becoming feasible:

  • Packaging/device cost barriers persist because contract manufacturers can’t achieve high yield without:
    • enormous capital expenses
    • known-good-die inputs

The proposed solution is end-to-end—not just better devices:

  • design-for-manufacturing flow from the start
  • clear ownership of the full stack (design → test → packaging → rack integration)
  • software layers to configure systems, diagnose faults, and remediate issues

Speakers stress it’s not “Lego pieces”—each component must be engineered around packaging and optical/rack integration processes.

A hyperscaler-driven economics lesson from scale-out is also referenced: pressure was used to drive cost/efficiency and stabilize the ecosystem (including the implication that some companies failed during earlier maturation—risk management matters in scale-up).

Product/feature themes mentioned (system requirements)

  • CPO insertion points: switch ASIC → compute packages → optical CXL memory
  • Optical engine needs: density via coupling technologies (e.g., rings/microrings), pluggable lasers, modular physical standards (detachable connectors)
  • Software-defined optics: configurable links plus deep telemetry for monitoring and maintenance
  • Rack integration cost mitigation: software diagnostics/remediation to prevent costly rack-level failures (e.g., fiber dust causing widespread failure)

Main speakers / sources

  • Timothy Pricket Morgan — host; “Next Platform”; sports writer for data center / composability focus
  • Vladimir Strianovich — CTO & co-founder, Lightmatter; optical IO / co-packaged optics R&D
  • Arez Shazaf — CTO, Alchip; AI accelerator I/O bandwidth bottleneck perspective
  • Adit Narimha — leads emerging technologies business group, Astera Labs; scale-up rack switch interconnects; interest in optically enabled roadmaps

Original video