Video summary

Scott Jenson: Are we really going to use the same Desktop UX forever?

Main summary

Key takeaways

Technology

Overview

Scott Jenson argues that desktop UX hasn’t meaningfully advanced in ~20 years, even as desktop computers have become far more capable. He frames the problem as an open-source leadership and UX design capacity issue: open source is busy and understaffed, and the Linux ecosystem has often waited for Apple/Microsoft to take risks instead of innovating itself.

Key themes and examples (desktop UX details)

Small UI details that matter

  • Ellipses placement

    • Jenson pushed for ellipses in the middle to preserve information.
    • Apple later implemented the approach.

  • Drag behavior across window focus (Mac vs Linux)

    • On many Linux systems, clicking (mouse down) brings a background window forward immediately.
    • On macOS, selection happens on mouse down, but the window comes forward on mouse up, enabling smoother “drag from background to foreground.”
    • Jenson emphasizes this is not just icon copying—it’s about directing user data movement into applications.
    • He credits a KDE contributor for implementing something similar and says he’s tried to meet them for ~2 years.

Desktop stagnation

  • Desktop UX models stabilized early and haven’t adapted well to modern needs, larger screens, and changed workflows.
  • He connects this stagnation to copying other platforms and not questioning assumptions, resulting in UX that becomes “historical cruft.”
  • He notes historical influence flows like Xerox → Mac → later cross-pollination, which helped establish patterns that then ossified.

Critique of common objections (“mobile won” / “standards are fixed”)

Jenson pushes back on three objections:

  1. “Do what mobile does”

    • Mobile won many markets, but not productivity.
    • Desktops remain uniquely strong due to big screens + keyboards.

  2. “It’s a standard, stop trying”

    • Past “standards” (e.g., CD-ROM, BlackBerry) faded.
    • Hardware and interaction models can and should evolve.

  3. “Don’t touch my stuff”

    • Users resist change, but Jenson argues for gentle, incremental evolution, not endless patching and bug-fixing forever.

UX framing: what “UX” actually includes

Jenson criticizes the vague/overused “UI/UX” framing and claims UX is misunderstood. He proposes UX as layered:

  • Style: visual/presentation (pixels, spacing, icons, colors)
  • Structure: navigation, flow, error handling, starting points
  • Strategy: who the user is, what matters, what to ignore
    • Also helps teams “say no” and prioritize
  • “Stuff” / underlying tech constraints: legacy limitations and historical assumptions

“Curse of direct manipulation” (and why desktop needs working-memory support)

Jenson argues that direct manipulation is powerful but stateless:

  • It’s great for rapid interactions.
  • But it breaks when users return later (next day/week): they struggle to remember what happened.
  • Clipboard/history limitations can cause context loss.
  • Linux users often mitigate this with many separate tools (window managers, shells, clipboard managers), frequently siloed from one another.

Proposed direction: working memory for desktop UX

Core proposal: improve desktop UX by designing for working memory across windows, clipboard, files, and history, so work context persists and can be revisited.

Historical / academic inspiration mentioned

  • “Life Streams” (1990s)

    • Organizes user data along a timeline.
    • Influenced WinFS, described as a relational-database-like file system history.

  • Nepomuk and similar projects

    • Jenson says they failed due to hardware/system constraints of the time, not because the vision was wrong.
    • He believes the ideas should be reconsidered with modern hardware.

AI on the desktop (where he sees gaps)

Jenson avoids making the talk “AI-centric,” but discusses how AI is being integrated:

  • Google AI pointer

    • Tries to keep desktop interaction largely unchanged while adding AI.

  • “AI OS” critique

    • Some systems feel like scripting/automation glued to chat, not genuine OS-level redesign.

  • Missed opportunity

    • He says most AI efforts focus on understanding the screen (on-screen awareness / assist structures).
    • They miss integrating the middle layer: the desktop’s working memory and context.

Claude Code / context over generation

He highlights that Claude Code improved because it used the file system for context, rather than relying purely on prompt context.

  • Framing question: can “context from the actual desktop” be expanded further?

Desktop “building blocks” he names

  • Windows
  • Files & icons
  • Clipboard
  • Mouse/keyboard as input systems

He emphasizes desktops fundamentally help users capture, move, edit, and store data—the basis of direct manipulation.

Prototypes he describes (concrete UX experiments)

Half the talk is presented as prototypes/ideas:

  1. Wide-screen desktop prototype

    • Reacts to how traditional Exposé scales poorly with many windows (becoming unusable).
    • Moves UI elements (e.g., menu bar) toward the center using a center/periphery focus concept.
    • Windows can be dragged/docked into side areas and can “shrink” into widgets (e.g., a music player becomes play/pause).

  2. Clipboard-as-document (persistent “collected stuff”)

    • Inspired by Obsidian: dragging items/text into a region makes them part of a document-bound clipboard/history container.
    • Persists across sessions (e.g., you close it and come back tomorrow—it’s still there).
    • Suggests local AI could later organize content (e.g., hotels → organize/map output).
    • Content can be edited and moved like a “baby virtual desktop.”

  3. Privacy-preserving telemetry concept

    • To avoid heavy personal data collection, he proposes collecting attention signals instead of content:
      • scroll duration, copy/paste events, browsing time

    • Uses simple math to reduce many signals (e.g., 50 browsing nodes → condensed to ~7 representative nodes).

    • Goal: reconstruct a useful history story locally to answer context questions like “Where did this map come from?” without exposing sensitive text.

“Three memories” model (organizing concept)

Jenson proposes exploring UX ideas through three memory types:

  • Spatial memory: messy organic organization of windows (wide-screen side/widget approach)
  • Associative memory: keeping data collected together and surviving restarts (document-bound clipboard)
  • Episodic memory: privacy-preserving activity/history narrative (timeline/story + provenance linking)

Open-source approach he recommends

  • Build experiments as optional, open-source prototypes.
  • Mentions the “In can switch” model: work with small teams to prototype quickly and open-source results.
  • He encourages Linux distros to try these ideas because “no one else is going to try and someone’s got to start.”

Review/question: privacy vs Windows Recall

A question asks how this avoids privacy failures like Windows Recall. Jenson responds:

  • The danger is becoming a “honeypot,” so the telemetry concept avoids collecting sensitive content.
  • Privacy needs serious discussion; next steps could include encryption and safer storage designs.
  • He argues the problem isn’t fundamentally unsolvable if implemented carefully.

Main speakers / sources

  • Speaker: Scott Jenson

    • UX designer; mentions prior work at Symbian, the Google/Chrome team, and the Apple/Finder team.
    • Current open-source UX work includes Mastodon and Home Assistant.

  • Referenced authors/projects (sources): Alan Kay, Donald Knuth (premature optimization quote), Don Norman (UX terminology evolution), “Life Streams” paper, WinFS, Nepomuk, Spotlight/Baloo, Claude Code, Google AI pointer, Windows Recall, KDE.

Original video