How to save money with Gemini Context Caching

Main Idea: Gemini Context Caching (money + latency savings)

Google introduced context caching for Gemini 1.5 Pro and Gemini 1.5 Flash.

Problem Addressed

Long context windows are expensive and slow. Historically, these windows were around ~1M tokens (and have since grown to ~2M tokens, potentially more). Large inputs—like big documents, codebases, or even video-length token counts—must be processed repeatedly.

How Context Caching Works

• First call: Gemini computes the large prefix/context (up to ~million tokens, plus any additional “tail” tokens you add).
• Subsequent calls: Gemini reuses the cached prefix instead of recomputing it.
• Pricing impact:
  • You pay for caching/storage,
  • but you avoid paying the full cost of recomputing the entire input workload each time.
• Performance impact:
  • The system only computes the newly added tokens beyond what’s cached.

---

Best Use Cases Highlighted

• Multiple queries over the same huge input, such as:
  • Video/audio files (multimodal)
  • Large document sets / books
• Preloading long system prompts for repeated tasks:
  • Useful for customer support-style chat workflows or any process that relies on a long, mostly-static system instruction.
  • Example: real products often use 10k+ token system prompts, especially with in-context learning patterns.
• Related concept: similar to prefix caching in other LLM frameworks, but Google provides user-facing controls so you can choose what gets cached.

---

Tutorial / Code Walkthrough (Implementation Details)

An example demonstrates loading a video and asking questions about it (e.g., extracting answers with timestamps).

Steps in the Demo

1. Import/enable caching support in the SDK.
2. Upload the multimedia file
   • Uses a Files-API-like flow.
   • The file persists for 24 hours.
   • The script waits for processing completion and then retrieves a file link.
3. Create cached content with cache_content_create
   • Choose the model: Gemini 1.5 Flash (also supports Pro).
   • Provide system instructions (acting as a long system prompt).
   • Include the uploaded video file as cached content.
   • Set cache duration/expiry (example uses 10 minutes; notes a default may be ~1 hour).
4. Run the query using a model configured with cached content
   • The first call includes full processing cost (very large token counts).
5. Run subsequent queries
   • Use model.generate_content.
   • Avoid redefining the cached wrapper for each query.
   • Outputs indicate that a large fraction of tokens are now cached content tokens—which correlates with faster responses and lower effective cost.

Metrics Observed in the Demo

• Without caching: prompt token counts were very high (reported around ~700k).
• With caching: logs include a cached content token count.
• Speed: noticeable improvement (example mentions roughly ~half a minute faster), with potentially larger gains when fewer tokens need recomputation (i.e., a larger cached fraction).

---

Main Speakers / Sources

• Primary source/speaker: The video creator (not named in the subtitles).
• Product/source referenced: Google Gemini and a Google IO announcement.
• The example code is based on an official Google sample.