Getting Evals Right for LLM Applications | Interrupt 26

Main technological focus

• The talk argues that LLM application “evals” (evaluation) should be treated like data science, not just software testing or “prompt-and-hope” grading.
• It contrasts prior ML/data science practice (e.g., careful data inspection, human label alignment, thoughtful metric design) with newer AI engineering habits (e.g., relying on “vibes,” using one model to grade another, and using generic/off-the-shelf metrics).

Key product/engineering ideas: harnesses & observability

• Example reference: OpenAI’s internal use of autonomous agents for months, supported by a “harness” that keeps agents on track.
• The harness includes logs, metrics, and traces—positioned as an observability stack.
• The presenter frames harnessing/evals as heavily dependent on data science practices, including:
  • error analysis
  • validation
  • metric design
  • data curation
  • monitoring

---

Common eval mistakes (and how to correct them)

The speakers highlight recurring pitfalls seen across teaching/evals at 4,500+ people/companies.

1) Using generic/off-the-shelf metrics

• Teams often use ambiguous metrics like “helpfulness,” “hallucination,” or “coherence.”
• Problems:
  • Definitions are unclear (e.g., what exactly counts as “hallucination”?).
  • Metrics differ across domains (e.g., medical vs. legal hallucination criteria).
• Recommended data-science approach:
  • Inspect real traces and data to identify specific failure modes.
  • Use AI-assisted tools to analyze traces message-by-message (examples mentioned: Codex, Claude Code, Cursor).
  • Create failure-mode catalogs and categorize into bespoke failure modes (example: a real-estate agent rescheduling tours due to hallucinated times).

2) Blind trust in LLM judges (without validating the judge)

• Pitfall: using an LLM judge to estimate failure frequency or grade outputs without checking alignment with humans.
• Common pattern:
  • Asking an LLM to rate outputs on a 1–5 or 1–100 scale
  • Then using histograms as “business metrics” (argued to be often not trustworthy or actionable)
• Recommended approach:
  • Treat the LLM judge like an imperfect classifier.
  • Build train/dev/test splits using labeled traces for each failure mode (same rigor as traditional ML).
  • Watch for imbalanced classification (failure modes may be rare).
  • Prefer precision/recall and false positives/false negatives over simple “accuracy.”

3) Bad experimental design—especially synthetic data problems

• Pitfall: synthetic data generated by LLMs becomes too uniform (synthetic traces look alike).
• Recommended approach:
  • Identify multiple varying dimensions of user inputs/contexts.
  • Use LLMs to vary each dimension, then take the cross product to create diverse coverage.
  • Emphasize quality review and diversity checks for synthetic data.
• Guiding exercise: inspect real traces and define at least three user-varying dimensions (e.g., persona: novice vs. expert).

4) Metric design that isn’t interpretable/actionable

• Pitfall: dashboards built from scoring scales (1–5, 1–100) that are hard to use for decisions.
• Recommended approach:
  • Convert to binary classification where possible (pass/fail).
  • Narrow judge scope to binary checks, and label data to measure alignment.
  • Accept that good judge prompting often requires iteration.

5) Outsourcing labeling and not trusting labels

• Pitfall: letting AI engineers/developers label data by default.
• Issue: unless it’s a coding app, they may lack domain expertise.
• Recommended approach:
  • Ensure labelers have domain expertise.
  • Always inspect labels and the underlying data, since unexpected failure causes may exist.

6) Criteria drift (rubrics don’t hold without data review)

• The talk references work on validating validators, introducing criteria drift:
  • People often don’t know what they want until they see examples.
  • Writing a rubric up front without data review is insufficient.
• Implication: rubrics/judges must be calibrated through data-driven review.

7) Over-automation of evals (letting LLMs do everything)

• Pitfall: assuming Claude/LLMs can “just do it.”
• Recommended stance:
  • LLMs can catch obvious errors
  • But deeper product failures require human-understood nuance and explicit context externalization

Additional eval anti-patterns mentioned

• Misusing similarity scores (e.g., ROUGE/BLEU) where similarity doesn’t map to real task success.
• Asking judges questions like “Is this helpful?” with vague prompts not grounded in product criteria.
• For annotation UX: don’t make annotators read raw JSON; build interfaces that reduce friction.
• Reporting uncalibrated judge scores without measuring alignment to humans.
• Sampling data effectively and ensuring dashboard metrics have actual signal.
• Avoid overfitting judges by repeatedly tuning/promoting based only on one dataset (set aside data, ensure generalization).

---

The proposed “data science skills” to improve evals

The closing summary maps eval improvements to classic data science capabilities:

• Error analysis / exploratory data analysis (EDA) on traces
• Metric design scoped to real failures and business outcomes
• Validation to ensure LLM judges align with human judgment
• Data curation and test-set discipline
• Monitoring & observability (logs/metrics/traces)
• Maintain a scientific mindset: experiment, measure, iterate

---

Slides / resources

• Slides are provided via a QR code (with mention of also sharing memes).

---

Main speakers / sources

Speakers

• Hamel (main organizer/contributor)
• Shreya (leads the “five big mistakes” section)

Referenced sources

• An OpenAI article about autonomous agents and a “harness” with logs/metrics/traces
• A paper on validating validators and criteria drift

Tools mentioned

• Codex, Claude Code, Cursor (for building interfaces to inspect traces)