Video summary

LHC Just Abandoned Red Hat For Debian Linux

Main summary

Key takeaways

Technology

Summary of technological concepts & product/engineering decisions

CERN’s Linux transition away from RHEL-adjacent ecosystems

CERN historically used Red Hat–based distributions, evolving through a few major community and downstream options:

  • RHEL 5/6 era
    • Scientific Linux (community-driven; led by Fermilab, CERN, DESY, ETH Zurich)
    • Status: now discontinued
  • RHEL 7/8 era
    • CentOS, later challenged by CentOS Stream

The talk frames the key issue: CentOS vs CentOS Stream differ fundamentally.

  • CentOS
    • Essentially “unbranded RHEL”
    • Removes RHEL branding while staying very close to RHEL release behavior
  • CentOS Stream
    • “Rolling source” upstream of RHEL
    • Introduces changes earlier—potentially before they land in a specific RHEL point release

Why Debian (specifically) became attractive

CERN runs a highly specialized, safety/uptime-critical environment with:

  • Custom hardware
  • Specialized drivers

The migration cost is high because everything must be verified to avoid breaking system behavior. At the same time, CERN can’t remain indefinitely on end-of-life (EOL) distros.

So the plan had to satisfy both:

  • Hardware stability constraints
  • Lifecycle/support realities

The “straw that broke the camel’s back”: CPU baseline changes

According to the speaker, CERN was forced to confront CPU microarchitecture baseline decisions in RHEL-like ecosystems via the compiler setting:

  • -march

A key example discussed:

  • x86-64-v2 → x86-64-v3
  • Implications include boot compatibility concerns

This was especially painful for CERN due to the presence of older, still-usable hardware.

Hardware impact estimates mentioned:

  • RHEL 9 baseline v2: reportedly 47% of hardware would stop functioning
  • RHEL 10 baseline v3: an additional 17%
  • Total: 64% no longer working

Beyond CPU compatibility, the speaker notes real-world upgrade costs such as:

  • Extensive testing/verification
  • Slot availability issues (may require new machines/rack space/buildings/permits)
  • Replacement of custom boards, implying:
    • new PCB design
    • additional engineering work

Timeline constraints for the migration

CERN’s accelerator runs ~24/7, so maintenance/upgrade windows mostly occur only during long shutdown periods.

  • Migration deadline discussed: Q4 2026
    • Rationale: next long shutdown for major OS upgrades

Debian release cycle conflicts create scheduling risk:

  • Debian Bookworm goes EOL before CERN’s full transition window completes
  • Later releases (Trixie, Forky) also have EOL timing that could otherwise strand CERN into additional risk/downtime

Using Debian support contracts / extended LTS to make the schedule work

The speaker emphasizes that, while Debian enterprises may use official repos, support vendors offer extended LTS windows.

Example given:

  • Freexian (specialized Debian support)

A proposed viability strategy:

  1. Move to Trixie for coverage through/into the operational window
  2. Potentially roll forward to Forky
    • So the support window spans successive shutdown periods
    • Avoids leaving systems EOL for too long

Core point: with extended LTS support, the Debian migration becomes feasible inside CERN’s multi-year shutdown planning.

Challenges of running Debian in this environment (packaging/tooling)

The talk highlights pain points in packaging/build workflows:

  • Lack of standard tooling for automated build + publishing tailored to CERN’s needs
  • Debian’s upstream tooling exists, but is not straightforward to deploy for CERN’s custom workflow

A promising solution mentioned:

  • Debusine (originating from Freexian)
    • Explored as a way to build and manage CERN’s customized package pipeline / custom distro building

Additional issues mentioned:

  • Binary kernel modules policy
    • CERN had to invent/define their own approach because Debian lacked an adequate policy for their needs
  • Difficulty in Debian workflows for running multiple versions of the same package/software concurrently
    • This is sometimes required at CERN

What’s emphasized as the “review/guide/tutorial” value

The content is framed less as a step-by-step tutorial and more as an analysis/engineering case study, focusing on:

  • Rationale for distro switching
  • Lifecycle/scheduling constraints (Debian release EOL vs shutdown windows)
  • Estimated risk and compatibility impact
  • Practical operational issues (testing cost, hardware constraints)
  • Migration feasibility via extended support contracts
  • Tooling gaps and mitigations (notably Debusine)

Main speakers / sources mentioned

Source organizations involved in prior distro leadership/history

  • Fermilab
  • CERN
  • DESY
  • ETH Zurich

These are cited in the context of Scientific Linux leadership/partner roles.

Support/vendor mentioned

  • Freexian
    • Extended Debian LTS support
    • Also linked to Debusine

Key entities/products discussed

  • Red Hat / RHEL
  • Scientific Linux
  • CentOS
  • CentOS Stream
  • Debian releases:
    • Bookworm
    • Trixie
    • Forky

Likely speaker

  • An unnamed YouTube narrator (no personal name provided in the subtitles)
  • Context: references a mini-DebConf in Switzerland talk (full talk ~48 minutes)

Original video