Video summary

Context Rot: How Increasing Input Tokens Impacts LLM Performance

Main summary

Key takeaways

Technology

Main theme

  • Context rot / long-context degradation: Even models with very large context windows (e.g., Gemini/GPT up to ~1M tokens, “Alma 4” up to ~10M) can produce unreliable results as input length increases. Performance is not uniform across the context window.

Why long-context benchmarks can be misleading

  • The well-known “needle” benchmark can appear strong because the task is relatively easy:

    • Lexical matching often solves it (the query is crafted so the “needle” is likely to match text patterns).

  • When evaluations add harder conditions, performance drops:

    • Ambiguity (similar needles, distractors)
    • Distractors (topically related but incorrect supporting text)
    • Longer conversation/memory retrieval

Key findings (experiments + results)

  1. Long conversational memory degrades with long inputs

    • Use case: chat assistant with “memory” across sessions.
    • Naive method tested: paste the full multi-session chat history into the prompt.
    • Evaluation: LongEval benchmark (≈ 500 messages per prompt; question at the end).
    • Comparison:
      • Full context: ~120k tokens average
      • Condensed context: only relevant snippets, ~300 tokens
    • Result: models perform better on condensed inputs, indicating degradation from extra context/noise.

  2. Ambiguity compounds the problem

    • Task: modified needle-in-a-haystack where the “needle” becomes more ambiguous.
    • Ambiguity is quantified via cosine similarity between the needle and the question.
    • Observation:
      • At short inputs, models can still handle high ambiguity
      • As input length grows, performance declines much faster with increasing ambiguity
    • Takeaway: models may “know” ambiguity resolution exists, but it breaks down with long contexts.

  3. Distractors hurt more at long lengths

    • Distractor definition: semantically/topically similar text that is not actually the correct answer.
    • Example: two similar “writing advice” sentences where one answers the wrong question.
    • Observation:
      • At short inputs, models can disambiguate needle vs distractor
      • At long inputs, accuracy drops even though the underlying task is the same

  4. LLMs are not reliable “computing systems” for exact tasks over long contexts

    • Synthetic test: “repeat a string n times” with one unique word inserted at a specific position.
    • Scoring: Levenshtein distance (edit distance) vs gold reference.
    • Result:
      • Performance drops even around 500 words
      • Errors include repeating beyond the expected content or random generation
    • Conclusion: models don’t process context consistently/uniformly.

Practical guidance: “context engineering” is required

  • Core recommendation: You must engineer your context window rather than assuming “more tokens = better.”
  • Even if technically you can use up to ~1M tokens, optimal working context is usually much smaller.
  • Framed as an optimization problem:
    • Maximize relevant information
    • Minimize irrelevant/noisy context

Techniques mentioned

  • Summarization (for multi-step agents)
    • Periodically summarize earlier actions/thoughts into shorter memory.
  • Retrieval (for recurring knowledge like docs/tools)
    • Store knowledge in a vector database
    • Retrieve only what’s relevant at each step
    • Tradeoff: requires experimentation/investment, but is fast and cost-efficient

Bottom line

  • A huge context window does not guarantee reliability at maximum length.
  • Long-context tasks can fail even when tasks are simple.
  • Therefore: context window management and experimentation are essential.
  • Mentions a full technical report: research.trychroma.com

Main speaker/source

  • Kelly — Researcher at Chroma (primary source of the video content).

Original video