Most devs don’t understand how context windows work

Main Ideas / Concepts

• Coding agents are debated, but many complaints (e.g., “agents suck”) are framed as a common user error: not understanding the context window.
• The context window is described as the primary constraint for modern AI coding agents.

How the Context Window Works (Core Explanation)

• Context window = all tokens the LLM can “see” at once, including:
  • Input tokens (everything you send), such as:
    • the system prompt (model instructions)
    • user messages initiating tasks or conversation
    • prior conversation turns
  • Output tokens (the assistant’s generated text), often produced as a stream
• As the conversation grows, the input + output token history grows.
• Eventually, the model hits a hard maximum (set by the model provider).
  • If you exceed it, you’ll get an LLM provider error or the generation may stop mid-output.
• You can hit limits via:
  • very long single messages
  • uploading/transcribing enormous documents
  • providing huge images
  • generating extremely long outputs that overrun the window during generation

Token Limits: “Bigger Isn’t Always Better”

• Context sizes mentioned:
  • Claude Haiku 4.5: 200,000 tokens
  • another model: 2 million tokens (referenced in a comparison view)
  • Gemini 2.5 Pro: described as having very large context windows, but the speaker says it doesn’t guarantee better results
  • Qwen Math Plus: about 4,000 tokens (smaller/older models often have much smaller limits)
• Why limits exist:
  • Compute cost: more context increases memory/processing load
  • Performance degradation: more context can worsen model performance
• Key reason cited:
  • “Lost in the middle” (needle-in-a-haystack)
  • If important information is buried in the middle of a large context, the model may struggle to retrieve and use it.

“Lost in the Middle” and Its Implications for Coding

• The speaker notes attention tends to prioritize:
  • content near the beginning
  • content near the end
  • content in the middle is often deprioritized
• They connect this to human-like memory biases:
  • primacy bias (start is remembered better)
  • recency bias (end is remembered better)
• Practical takeaway for AI coding agents:
  • Keep important instructions and relevant code context near the start/end, or
  • reduce/refresh bloated context.

Methodology / Workflow Instructions (Practical Guidance)

1) Monitor Context Usage

• Use tooling (example shown with Claude Code) to check usage, e.g.:
  • “95k tokens out of 200k”

2) Know When to Reset

• Example threshold behavior described:
  • If there’s lots of free space, continue
  • If you have roughly ~50k tokens left, the speaker recommends running clear (reset chat history)

3) Clear vs. Compact (Claude Code-Specific)

• Clear
  • Deletes conversation history
  • Frees up context window by returning to a blank slate
  • Recommended as the default for a clean restart
• Compact
  • Replaces prior messages with a summary to preserve “intent/vibes/rules”
  • Benefits:
    • reduces context size
    • can reduce “lost in the middle” issues by shortening the active prompt
  • Tradeoff:
    • uses tokens/time because the LLM summarizes
    • may delay completion (the speaker estimates about a minute)

4) Keep Context Lean by Avoiding Bloated Components

• Word of warning: MCP servers
  • MCP servers are attractive for “plug and play” tool integrations
  • But they can bloat context rapidly
  • The speaker recommends being cautious about adding many MCP servers

5) Avoid Overly Long Rule Prompts

• The speaker tries not to write very large “Cursor rules” or “Claude rules” because:
  • long rules increase context size
  • can worsen lost-in-the-middle effects

6) Evaluate Retrieval, Not Just Context Size

• Don’t only check numeric context length—evaluate:
  • how well the model retrieves information from its context window
• Example cited:
  • Meta announced Llama 4 Scout with a 10 million context window
  • but users reported poor lost-in-the-middle performance, so huge context didn’t help much.

Key Lessons / Takeaways (Condensed)

• Many “coding agent” failures may be context-window issues, not agent capability alone.
• Context window management is essential:
  • monitor token usage
  • clear when nearing limits
  • compact if you want to preserve conversation intent
  • keep tool integrations and rules lean to prevent lost context
• Longer context windows can still perform worse due to lost-in-the-middle retrieval problems.
• When assessing models, consider retrieval quality, not just max tokens.

Speakers / Sources Featured

Speakers

• The video speaker/creator (name not provided in subtitles; referred to as “I” throughout)

Tools / Systems Mentioned

• Claude Code
• Claude (e.g., “Claude Haiku 4.5”, “Sonnet 4.5”)
• ChatGPT
• Gemini 2.5 Pro
• Qwen Math Plus
• Meta Llama 4 Scout

Web / Resources Mentioned

• models.dev
• aihero.dev (for an “AI SDK crash course”)
• Mentioned: an alternate video on tokens (link referenced but not provided in subtitles)

Other Concept Sources

• MCP (Model Context Protocol) servers

Organizations / Platforms

• Vercel (AI SDK crash course)