Video summary

Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind

Main summary

Key takeaways

Technology

Summary

Philipp Schmid (Google DeepMind) argues that skills (tool modules the model can invoke) should never be shipped without evaluation (evals). In production, failures are difficult to attribute when agent behavior is non-deterministic, and skills may be unused, over-triggered, under-triggered, or simply unnecessary for newer models.

1) Why evals are needed

In production, it’s often unclear whether a task failure is due to:

  • the skill being bad, or
  • the task being too hard for the model.

Schmid cites Skill Bench v1.1, an analysis of ~50,000 skills, finding almost none had evals. Many skills were AI-generated but not tested.

Because agents are non-deterministic, reliability must be measured across runs.

2) “Agents we use” vs “agents we build” (context matters)

Schmid distinguishes between:

  • External coding/productivity agents (examples mentioned): Cursor, Claude Code, (anti-gravity)

    • These agents may trigger skills implicitly.
    • Users quickly notice when skills don’t work (e.g., reprompting, slash commands).

  • Built agents for customers

    • These generally don’t know what a “skill” is.
    • The customer-facing agent may not explicitly instruct the model to use the right skill (e.g., “refund skill” vs “help me with support”).

This distinction affects how skill triggering works and how evals should be designed.

3) What a “skill” is, and how it works

A skill is described as:

  • A folder containing a skills.md plus assets/files.

It uses progressive disclosure:

  1. Title + description
    • Added to model context
    • Used to decide whether to trigger
  2. Skills body
    • More instructions/details
  3. Reference files
    • Deeper, task-specific context

4) Two types of skills

Capability skills

  • Teach the model something it can’t reliably do yet (e.g., log tracing, creating a React app template).
  • These are temporary and should be retired when models improve.
  • Evals determine when it’s safe to retire.

Preference skills

  • Encode team-specific workflows, style, language preferences, or domain references.
  • More durable; should be protected so agent updates don’t degrade performance.

5) Skills can improve performance—if validated

  • Skill Bench suggests skills improve performance by roughly ~15% on average.
  • For AI-generated skills, he reports:
    • Human-written skills performed best
    • AI-generated skills may harm performance
  • Rough guideline: keep skill files under ~500 lines of words
    • Too long increases cost and can reduce effective behavior.

6) Triggering modes: model-invoked vs user-invoked

Model-triggered skills

  • The model decides to read/trigger based on context and description.

User-invoked skills

  • The system/user explicitly requests/uses them (e.g., create PR, stage documentation).
  • Schmid recommends many “script-like” dev tasks should be user-invoked rather than consuming tokens for model-trigger discovery.

His eval guidance focuses primarily on model-invoked skills.

7) How to write good skills (best practices)

Schmid planned “8 examples/tips” and later reiterates “10 best practices.” Key points include:

Skill description is critical

  • The description is the main trigger signal (often two sentences).
  • If weak, it can cause:
    • over-triggering
    • under-triggering
  • Good descriptions define:
    • why to use the skill
    • how the model should use it
    • when to use it

Prefer directives over essays

  • Use explicit instructions, e.g. “Use Interactions API when building chat/web interaction apps,” rather than passive explanation.

Keep skills lean; layer info

  • Description cost is paid on every model call (token cost).
  • Put deep details in reference files, not in skills.md.

Avoid over-constraining workflow steps

  • If the process is always the same (fixed path), a script is better than a skill.
  • Define goals/constraints, not rigid step-by-step procedures.

Include negative cases (“when NOT to use”)

  • Add prompts where the skill should not trigger to prevent broad matches (e.g., don’t use a generic “web dev” skill when you only want React/Tailwind).

Test early

  • Start with ~10–20 prompts:
    • ~5 “happy path” triggers
    • ~5 “not trigger” cases
  • Include real customer/production traces when available.

Remove “no-ops”

  • AI-generated skills often include no-ops (instructions that don’t change behavior).
  • These waste tokens and cost even if eval performance doesn’t change.

Retire skills when possible

  • Run evals with and without the skill enabled.
  • If performance holds without triggering it:
    • retire the skill to save cost and maintenance.

8) Practical eval harness example: Gemini Interactions API skill

Schmid provides a case study:

  • Goal: create a skill to help models generate valid code for the Gemini Interactions API
    • The API was released after the last Gemini training run, so older model versions lacked context.
  • Test suite: 117 test cases
    • Built from real user patterns (synthetic and inferred from telemetry/feedback)
    • Included feedback like: models use older Gemini patterns (e.g., 2.0) even when using newer models (e.g., 3.0)
  • Result: improved performance up to ~90% for generating valid Interactions API code with latest models.

Test harness design

Only two simple assets were needed:

  • A JSON test-case file with fields such as:
    • prompt
    • language (TypeScript/Python)
    • should_trigger boolean
    • expected checks (regex/assertions)
  • A Python script that runs a coding agent (e.g., Gemini CLI) and validates outputs.

He notes many skill evals can use regex-based cheap assertions (often no “LLM as judge” required). More complex skills use LLM-as-a-judge with rubrics.

Regression gating

At DeepMind:

  • evals live alongside every skill
  • evals run on every skill change (diff gating):
    • merges happen only if they improve test cases
    • otherwise, failures block merging

9) Additional eval guidance (quality controls)

  • Test outcomes, not just whether the skill loads. A skill triggering after several turns can still be acceptable.

  • Use isolated runs to prevent “cheating” via prior context.

  • Run multiple trials per case (due to non-determinism; “to six trials” mentioned).
  • Evaluate across different harnesses/models (e.g., Cursor vs other agents) to match real customer setups.
  • Keep evals even when retiring a skill:
    • detect performance degradation
    • determine when to reintroduce it

Core takeaway

“Don’t ship skills without evals.” Evals ensure correct triggering, prevent regressions, quantify reliability, control cost, and support safe skill retirement.

Main speakers / sources

  • Speaker: Philipp Schmid (Google DeepMind)
  • Referenced resource: Skill Bench v1.1
  • Referenced educator/tooling source: Matt (AI educator; referenced for “no-ops” insight and a skills repository)

Original video