Prompt Caching Explained: Stop Overpaying for AI Agents

Key idea: Prompt caching to reduce LLM agent costs

In agent chats, the model typically reprocesses the entire conversation context each turn. That means if you start with large prior context (e.g., 50k tokens) and then add more (e.g., 1k tokens), the model often ends up processing:

• 51k tokens on the next turn
• 55k tokens on the following turn

As a result, costs can grow quickly when the full history is resent and re-evaluated.

Prompt caching reduces this by reusing repeated prompt content at a lower effective rate—for example, the first time is full price, while later repeats may be closer to ~10% price (as described).

---

Misconception debunked: caching isn’t like database query caching

A common mental model—“cache the prompt output and reuse it when the same prompt repeats”—is described as wrong for prompt caching.

Instead, this caching is effectively about reusing/caching the input prompt content, not “raw data” outputs in the way people expect from traditional database caches.

---

How agent context drives repeated token costs

A typical flow looks like this:

1. Start with 50,000 context tokens
2. Add 1,000 new tokens (new question/request) → 51,000 total
3. Next turn resends expanded history (e.g., 55,000)

Even if an API (like the Responses API) feels like the model remembers, the speaker claims the provider still processes the full context each time behind the scenes.

---

Provider pricing behavior (why caching matters)

Many providers price tokens differently depending on whether content has been seen before:

• First occurrence: higher cost
• Subsequent occurrences: cheaper due to caching

The speaker compares major providers (e.g., OpenAI / Anthropic / Google / DeepSeek, plus others) and emphasizes that:

• Without caching, longer sessions can be very expensive (they mention extremely high costs for ~200k-token sessions).
• With caching, costs become much more linear as session context grows.

They highlight DeepSeek as being “almost free” in a hypothetical comparison (with the caveat that prices can change).

---

Best practices / guide for implementing prompt caching correctly

1. Enable prompt caching properly for your provider + API

• Some providers using Responses API may enable caching by default.
• With Chat Completions, caching may require explicit configuration—verify in your setup.

2. Check cache hit rates and tracking

• Use tooling/agent frameworks that show cache hit rates (hits per request/session).
• In the demo tooling (Tau/Pi), sidebar metrics show cache performance.

3. Account for cache expiration

Cache entries may expire after a provider-specific window, for example (as described):

• OpenAI: ~1 hour (in their setup)
• Anthropic: ~5 minutes by default, and ~1 hour with certain auth setups

When cache “runs out,” you may need to re-warm later—meaning parts of the prompt get priced again at full rates.

4. Know whether the provider caches prompts automatically

The speaker claims caching behavior varies:

• Some providers (e.g., OpenAI, and certain setups like Hugging Face inference) may cache input automatically
• Others (e.g., Anthropic, Gemini) may not, requiring enabling features in the client/agent

5. Avoid invalidating cache with dynamic system prompts

Don’t place changing data in the system prompt, such as:

• timestamps
• current working directory
• dynamically updated tool lists

If the system prompt changes, cached input may be invalidated—raising costs.

6. Be aware that summarization/compaction can reset cache

If conversation history is compressed into a summary and older content is replaced, cached segments may be invalidated. This is expected behavior.

---

Practical demonstration notes (tooling)

The speaker demonstrates using Tau (Python port of Pi), including:

• cache hit rates in the UI
• exporting session transcripts to HTML
• showing large token “traffic” (e.g., 10.9 million tokens exchanged) while keeping cost low due to caching

They also note routing through the same provider to avoid losing cache (i.e., preserving “warm cache” behavior).

---

Main speakers / sources

• Main speaker: Unspecified individual narrator/presenter (no name provided in subtitles)
• Sources/products referenced: OpenAI, Anthropic, Google, Gemini, DeepSeek, Hugging Face Inference, Together AI, Cerebras, plus tooling frameworks Pi and Tau and APIs like Responses API / Chat Completions.