Summary of "The Internet Was Weeks Away From Disaster and No One Knew"

High-level story

In 2024 an elaborate supply-chain backdoor was discovered in the XZ compression project. XZ (LZMA) is a widely used lossless compressor and a dependency in many Linux distributions and in the OpenSSH toolchain.
A malicious contributor (alias “Jia Tan”) attempted to insert a stealthy backdoor that would give an attacker a master key to authenticate into SSH on virtually any affected Linux server.
The backdoor reached pre-release packages for distributions including Fedora, Debian testing and Ubuntu pre-release and might have been on a path into Red Hat Enterprise Linux (RHEL).
Detection began with a suspicious performance slowdown investigated by Andres Freund. After coordinated disclosure, Fedora rolled back the packages and the community removed the malicious code.

Cryptography and authentication
- Diffie–Hellman key agreement: how two parties establish a shared secret over an eavesdropped channel (illustrated as public‑color + private‑color paint mixing).
- RSA public‑key authentication: two secret primes generate a public modulus; the published public key authenticates whoever knows the factors.
- SSH: combines a secure channel (key agreement) with authentication (RSA) for secure remote logins; OpenSSH is critical infrastructure for remote server maintenance.
Compression algorithms
- Huffman coding: variable-length codes based on symbol frequency.
- LZ77: sliding-window pointers that copy repeated substrings.
- LZMA: Lempel–Ziv plus Markov modeling with a large search window; XZ uses LZMA to achieve better compression than typical .zip/deflate.
Shared libraries and runtime linking
- Global Offset Table (GOT): how applications resolve addresses of shared-library functions at load/run time.
- IFUNC resolver: lets a library pick different implementations early at runtime (for example, optimized code for different CPU vendors).
- Dynamic audit hooks: a low-level feature that can run code when the loader writes GOT entries.
Memory debugging
- Valgrind: detects memory errors (invalid writes, leaks) and can expose misbehaving or malicious components.

Trojan horse (supply‑chain entry)
- Hide malicious payload inside binary “test blobs” in the XZ repo (binary data files packagers commonly ignore).
- Make superficial benign changes on GitHub to build trust and be accepted as a contributor/maintainer.
Goldilocks timing (GOT hijack)
- Use an IFUNC resolver inside the XZ library to run code very early during program load.
- From the IFUNC, locate and set a dynamic audit hook so that when the loader writes the real address of a crypto function (RSA decryption) into the GOT, the audit hook runs and swaps the GOT entry with the attacker’s function implementation — timed between the real write and the subsequent read-only lock.
- This gives the attacker control over the authentication function used by OpenSSH without modifying OpenSSH source directly.
Cat burglar (stealth master key and cleanup)
- The malicious function checks for a special master-key exchange (its own mini‑cryptographic handshake) and only then grants access; otherwise it defers to the real function.
- The backdoor obfuscated strings, performed safety checks to avoid crashing, and attempted to wipe logs to hide traces.

The backdoor introduced a subtle latency (~400–500 ms), which tipped off a developer (Andres Freund). That timing anomaly led to code inspection, discovery of the backdoor, and rapid community response.
The demonstration in the video shows the exploit live (on a cloned Veritasium site) to illustrate that a compromised server could grant root control, modify website content, exfiltrate or alter data, or install ransomware.

Impact potential
- A universal SSH backdoor could enable spying, ransomware, or widespread disruption on millions of servers, including critical infrastructure (banks, governments, supercomputers).
Root causes identified
- Dependency concentration in the open-source ecosystem: many projects rely on small, volunteer‑maintained components (single‑maintainer risk).
- Social engineering and long multi‑year campaigns can coerce or replace maintainers to insert malicious code.
- Not all dependencies receive equal scrutiny, even when top-level projects (like OpenSSH) are well audited.
Response
- Debian, Red Hat and Fedora teams rolled back or blocked tainted packages after disclosure. The community audited related projects for similar compromises.
Broader debate
- The incident highlights both vulnerabilities in open-source supply chains and a key strength: public code allowed community review to find and remove the backdoor.
- It underscores the need for better funding, multiple maintainers, and stronger review/auditing guarantees for critical projects.
Attribution
- Attribution remains uncertain. Evidence (timezones, naming, operation patterns) is ambiguous and could be intentionally misleading. Defenders must assume sophisticated, persistent adversaries.

Compromised and involved projects
- XZ (LZMA compressor) — compromised project
- OpenSSH — target/application relying on XZ indirectly
- Fedora, Debian, Ubuntu, Red Hat Enterprise Linux (RHEL) — distribution ecosystems involved
Tools and primitives
- Git / GitHub — source hosting and contribution workflow
- Valgrind — memory error detector that helped expose problematic code
- Global Offset Table (GOT), IFUNC resolver, dynamic audit hooks — runtime linking/hijack primitives
- SSH, Diffie–Hellman, RSA — crypto/authentication building blocks
Other mentions / context
- Postgres — unrelated project where the discovering developer worked (Andres Freund)
- GNU Project, Free Software Foundation, Linux kernel and GNU tools — historical context for open source

Supply-chain security
- Account for subtle maintainer takeovers and hidden binary blobs inside repositories; code review should include test data and build artifacts.
Project sustainability
- Widely depended-on open-source components need sustainable funding, multiple maintainers, or stronger review/auditing guarantees to avoid single‑person failure modes.
Detection approaches
- Public/open code enables discovery and remediation, but attackers can design backdoors to survive heavy scrutiny. Detection may rely on side signals (performance anomalies) as much as code review.
Operational assumption
- Treat potential attackers as sophisticated and persistent; don’t assume obvious attribution.

Derek (Derek Muller) — Veritasium host / on-camera interviewer
Narrator (labeled in transcript as Narrator / Henry in parts)
Richard Stallman — history of Free Software / GNU
Linus Torvalds — origin of Linux (referenced)
Tatu Ylonen — creator of SSH (explains crypto)
Lasse Collin — original XZ maintainer (victim / single‑maintainer example)
Jia Tan — alias used by the attacker(s) (malicious contributor)
Rich — Fedora packager who accepted XZ update (packager perspective)
Andres Freund — Microsoft/Postgres developer who discovered the backdoor
Organizations quoted: Red Hat, Debian, Ubuntu, Microsoft, NTT, and the open-source community generally

Compact timeline of attack → discovery → mitigation steps.
Diagram of the attack flow: Trojan → IFUNC/audit hook → GOT swap → backdoor authentication.
Extracted technical mitigations with implementation pointers (CI checks, binary-blob scanning, reproducible builds, mandated multiple maintainers).