Video summary

10 Websites That Can Find What the Internet Deleted

Main summary

Key takeaways

Educational

Main ideas & lessons (what the video conveys)

  • The internet constantly “erases” itself through:
    • Server outages
    • Platform shutdowns
    • Corporate edits
    • Link rot
    • Technical barriers (e.g., anti-bot systems)
  • Internet archiving responds to “digital amnesia” by preserving websites and digital culture before they become inaccessible or untrustworthy.
  • Different archiving services solve different problems, including:
    • National/cultural legal preservation (government/library vaults)
    • Searchability of the dead web (finding content without exact URLs)
    • Machine-readable raw datasets (useful for AI training)
    • Handling modern dynamic platforms (including JS-heavy and anti-bot sites)
    • Legal integrity and citation stability (preventing link rot in courts/academia)
    • Self-hosted personal preservation (reducing dependence on third parties)
    • Preserving interactive behavior (not just page snapshots)
    • Cross-archive aggregation (reducing fragmentation across archives)
  • The Wayback Machine is presented as the most resilient baseline, but it still faces major cyberattacks and legal pressure.

Methodologies / approaches presented

UK Web Archive (Number 10) — national legal-deposit crawling

  • Treats web content like historically important documents rather than disposable media.
  • Built as a large-scale digital preservation project run by:
    • A consortium of six legal deposit libraries
    • Spearheaded by the British Library
  • Works by:
    • Crawling and collecting every UK website at least once per year
  • Captures a wide variety of content (e.g., government announcements, local disputes, fan pages).
  • Goal:
    • Preserve the nation’s digital “memory” so digital rot isn’t accepted as inevitable.

Library of Congress Web Archives (Number 9) — curated “surgical” preservation

  • Focuses on relevant cultural and governmental snapshots, not everything.
  • Operates by:
    • Maintaining over 180 event and thematic collections
    • Actively crawling around 15,000 specific seed URLs at a time
  • Examples of preserved targets:
    • Congressional records
    • National elections
    • Supreme Court nominations
    • Major cultural shifts
  • Goal:
    • Freeze key moments so future historians don’t face a “void” in early 21st-century evidence.

Archivo.pt (Number 8) — make the dead web searchable without exact links

  • Addresses the limitation of most archives that require the exact dead URL.
  • Operates as:
    • A Portuguese web archive (operating since 1996) preserving billions of files
    • A search engine for the past web using Google-like full text and image search
  • Approach:
    • Indexes text and images from long-dead HTML so users can search by:
      • Phrases
      • Historical events
      • Product names
      • Topics from specific time periods (e.g., 2004)
  • Goal:
    • Turn inaccessible digital “graveyards” into an easily navigable library.

Common Crawl (Number 7) — open raw crawling dataset for machines/AI

  • Operates as a nonprofit that:
    • Crawls the web
    • Scrapes data
    • Freely releases archives/data sets to the public
  • Timeline/scale:
    • Doing this since 2008
    • Accumulated ~10 pabytes of raw data (as stated)
  • Emphasis:
    • Optimized for raw text/data, not human-friendly browsing
  • Claimed impact:
    • Foundational dataset used to train many modern AI/large language models
  • Goal:
    • Serve as “raw material” powering algorithmic systems dominating the economy.

Ghost Archive (Number 6) — archiving dynamic, JS-heavy, anti-bot platforms

  • Targets a key modern problem: traditional crawlers often capture blank pages or loaders.
  • Designed to bypass limitations like:
    • Heavy JavaScript
    • Infinite scrolling
    • Anti-bot protections
  • Capabilities/constraints (as stated):
    • Captures pages up to 50 MB
    • Saves video from platforms like Reddit and YouTube up to 360p, under a 100 MB limit
  • Targets:
    • Social platforms where modern culture happens
    • Content types vulnerable to deletion (videos, channels, purged subreddits)
  • Goal:
    • Provide “anti-rot” preservation so modern social media history doesn’t vanish after site changes.

Perma.cc (Number 5) — preventing link rot for legal/academic citations

  • Problem/context:
    • Legal and scholarly citations break over time (“link rot”).
  • Claimed evidence of scale (as stated):
    • A 2024 Pew Research study found 38% of web pages from 2013 are inaccessible.
  • How it works:
    • Developed by Harvard Law School Library Innovation Lab
    • Creates a permanent, unalterable archived link
    • Captures a page snapshot at a precise moment (“that exact millisecond”)
    • Stores it in an institutional vault
  • Goal:
    • Ensure cited evidence remains stable and unchanged for courts, scholars, researchers, and journalists.

Archive Box (Number 4) — self-hosted personal archiving to avoid third-party risk

  • Core idea:
    • Don’t rely on third parties to preserve your specific content (to avoid loss from funding changes, lawsuits, server failures, or cyberattacks).
  • Approach:
    • Open-source, self-hosted web archiving system
    • Decentralizes preservation by running it on your own computer or server
  • Inputs:
    • Feed it individual URLs
    • Or connect it to browser history / bookmarks folders
  • Outputs/formats mentioned:
    • Saves local copies in standard readable formats such as HTML, PDF, WARC, and PNG
  • Goal:
    • Build your own “personal Wayback machine” on hardware you control.

Web Recorder (Number 3) — high-fidelity preservation of interactive sessions

  • Problem:
    • Many modern sites are interactive software; screenshots/crawlers can’t preserve functionality.
  • Approach:
    • Open-source tool focused on high-fidelity archiving
    • Records the user’s interactive session
  • What it records (as described):
    • User actions like dropdowns, map expansions, complex script executions, and navigation through 3D spaces
    • The exact behavior and network traffic in real time
  • Goal:
    • Preserve the experience so future users can “click through” the site as it actually worked.

Momento Time Travel (Number 2) — unify searching across many archives

  • Problem:
    • Archives are fragmented; finding a vanished page can become a time-consuming checklist across many providers.
  • Approach:
    • An aggregator using the Momento protocol (RFC7089) to add a standardized “time dimension” to web requests.
  • How users interact with it:
    • Provide a URL and a target date
  • What it does:
    • Searches multiple global web archives simultaneously
    • Cross-references major archives in a fraction of a second
  • Goal:
    • Bridge gaps in the archiving ecosystem so the timeline can be stitched together.

Wayback Machine (Number 1) — continuous large-scale public web capture and resilience

  • Role:
    • Presented as the “heavyweight champion” and baseline of digital history.
  • Approach:
    • Continuous operation since 1996
    • Methodically “photographs” the public web
  • Scale milestone (as stated):
    • Crossed 1 trillion archived web pages (October 2025)
    • Over 100 pabytes of historical data
  • Threats described:
    • Massive cyber attacks and DDoS campaigns (2024–2026 timeframe mentioned)
    • High-profile lawsuits from corporate publishers
  • Goal:
    • Preserve human digital culture despite attacks and legal pressure.

Overall takeaway

  • No single tool fully solves web preservation.
  • The video frames internet archiving as a spectrum of solutions spanning:
    • Legal custody
    • Curated historical capture
    • Search over archived content
    • Machine-friendly datasets
    • Dynamic-content capture
    • Citation-grade immutability
    • Self-hosted control
    • Interactivity recording
    • Cross-archive time-based aggregation
  • The Wayback Machine is positioned as central infrastructure, but all these tools together form a broader defense against the disappearance of digital history.

Speakers / sources featured

  • Sources named (organizations/tools):
    • UK Web Archive (consortium of six legal deposit libraries; British Library spearheading)
    • Library of Congress
    • archivo.pt
    • Common Crawl
    • Ghost Archive
    • perma.cc (Harvard Law School Library Innovation Lab)
    • Archive Box (open-source, self-hosted)
    • Web Recorder (open-source)
    • Momento Time Travel (Momento protocol RFC7089)
    • Wayback Machine (Internet Archive)
    • Pew Research (2024 study mentioned)
  • No individual human speakers are identified in the subtitles.

Original video