Video summary

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 8 - LLM Evaluation

Main summary

Key takeaways

Educational

Main ideas and lessons (LLM evaluation focus)

1) Why evaluation matters

  • The course frames LLM evaluation as essential: without measuring performance, it’s unclear what improvements to make.
  • The lecture emphasizes output quality evaluation, even though scoring free-form text is difficult (answers may be natural language, code, math reasoning, etc.).

2) Recap of prior methods that evaluation builds on

  • RAG (Retrieval-Augmented Generation): an LLM retrieves external knowledge from a knowledge base.
    • Two steps:
      • Candidate retrieval: typically a bi-encoder / embedding model (e.g., Sentence-BERT) to shortlist relevant passages.
      • Reranking: a more complex model (often cross-encoders) to refine candidate ordering.
    • Retrieval quality can be quantified.
  • Tool calling: the LLM decides which tool to call and with what arguments, then returns tool results in natural language.
  • Agentic workflows:
    • Combine RAG + tool calling, potentially with multiple tool calls per request.
    • Example framework: ReAct using Observe → Plan → Act.
  • The lecture links these to evaluation: agents and tools introduce new failure modes that must be measured.

Defining “evaluation” in this lecture

Evaluation can include many things (output quality, coherence, factuality, latency, pricing/uptime, etc.), but here:

  • Primary goal: quantify how good the model’s actual responses are.

Human evaluation: ideal but impractical

Ideal approach

For each prompt:

  1. Generate a response with the LLM.
  2. Have one or more humans rate the response.
  3. Aggregate ratings to estimate overall model performance.

Limitations addressed

  1. Subjectivity & inter-rater disagreement

    • Example: evaluating the usefulness of a prompt like “teddy bear gift” can vary:
      • One rater likes it.
      • Another criticizes missing specifics (e.g., which stuffed animal?).
    • Requires inter-rater agreement metrics to quantify consistency.

  2. Agreement metrics can be misleading without chance baselines

    • If raters guess randomly, they can still agree at non-trivial rates (e.g., ~50% for binary labels under equal randomness).
    • Therefore, metrics adjust observed agreement relative to chance (i.e., “agreement-by-chance”).

Inter-rater agreement metrics mentioned (chance-corrected)

  • Cohen’s kappa (explicitly referenced)
  • Fleiss’ kappa and Krippendorff’s alpha (mentioned as variants/extensions)

Core idea:

  • Measure how much better observed agreement is than random agreement.
  • Agreement = 1 → perfect agreement
  • Below chance → negative coefficient

Practical mitigation when humans disagree

  • Run alignment sessions (“agreement sessions”) between raters to standardize guidelines until inter-rater agreement stabilizes.

Rule-based metrics using references (no per-response human rating)

Approach

Instead of asking humans to rate each output:

  • Ask humans/experts to create reference outputs
  • Use automated rule-based similarity metrics comparing prediction vs reference

Benefits

  • Lower cost and faster iteration (reference set stays fixed)

Key challenge

  • Natural language is stylistically variable, so exact matching often fails.

Major translation/summarization metrics described

  1. METEOR (“Metric for Evaluation of Translation with Explicit Ordering”)

    • Structure: (weighted precision/recall harmonic-style score) × (1 − penalty)
    • Penalizes incorrect word order
    • Uses:
      • Hyperparameters (alpha, gamma, beta)
      • Concepts like contiguous chunk matches (favors fewer chunks for better ordering)
    • Limitations:
      • Feels “recipe-like” due to arbitrary hyperparameters
      • Limited handling of stylistic variation

  2. BLEU (“Bilingual Evaluation Understudy”)

    • Emphasizes matching n-grams (precision-like perspective)
    • Includes a brevity penalty to discourage overly short outputs
    • Limitation: correlation with human judgment is imperfect; style variability hurts scoring

  3. ROUGE

    • Common for summarization
    • Overlap-based variants similar in spirit to BLEU/METEOR

Two key limitations of reference-based rule metrics

  • Poor handling of paraphrasing / stylistic variation
  • Weak correlation with human ratings
    • Often needs human ratings initially for tuning/correlation.

Core method: “LLM as a Judge”

Concept

  • Use a separate LLM to evaluate a candidate LLM response.
  • The judge model receives:
    • the original prompt
    • the candidate response
    • the criteria/rubric for grading

Judge outputs

  • Typically returns:
    • Score (binary pass/fail or graded scale)
    • Rationale explaining why

Benefit vs rule metrics

  • Explainability: rationales are interpretable, not just opaque numbers.

Prompt ordering trick (to improve structure/quality)

  • Ask the judge to output rationale first, then score.
  • Motivation: encourages “externalized reasoning” prior to judgment.

Important limitation: outputs may not be parseable

  • LLM outputs are probabilistic:
    • A judge prompt may not consistently return both rationale and a parseable structured score.

Method to guarantee structured output format

  • Use constrained guided decoding / structured output
    • Require a specific format (e.g., JSON with rationale and score)
    • Providers (OpenAI/Gemini/Anthropic) offer “structured output” tooling.

Types of “LLM judge” setups

1) Pointwise (single response)

  • Question: “Is the response good or not good?” (pass/fail or similar)
  • Judge evaluates one response.

2) Pairwise (preference)

  • Provide two responses (A and B)
  • Ask: “Which response is better?”
  • Useful for:
    • generating synthetic preference data
    • training preference models (supporting preference tuning workflows mentioned earlier)

What can go wrong with LLM-as-judge (biases + mitigation)

Failure mode 1: Position bias

  • The judge may prefer the response that appears first.
  • Mitigation:
    • evaluate both orders:
      • “A better than B”
      • “B better than A”
    • use majority voting (or similar strategies)

Failure mode 2: Verbosity bias

  • The judge may prefer longer/more verbose answers even when they aren’t better.
  • Mitigations:
    • instruct the judge not to overvalue length
    • provide rubric examples showing verbosity isn’t automatically better
    • add explicit length penalties in the scoring prompt

Failure mode 3: Self-enhancement bias

  • If the judge evaluates outputs produced by itself (or a closely related model), it may favor its own style.
  • Mitigation:
    • use a different model for generation vs judging (ideally higher capacity or at least different instance)

Additional note

  • Judge misalignment with human preferences is possible.
  • A stronger judge (often “bigger” and more reasoning-capable) can reduce susceptibility to superficial cues.

Best practices for LLM-as-judge evaluation

  • Provide crisp guidelines:
    • define what “good” and “bad” mean
    • clarify boundaries to reduce ambiguity
  • Prefer binary scoring when possible:
    • pass/fail often reduces noise from multi-level subjectivity
  • Ask for rationale before score
  • Mitigate known biases (position/verbosity/self-enhancement) with the strategies above
  • Calibration to humans (when possible):
    • collect some human ratings + judge scores
    • run correlation analysis to see how well judge scores track human preferences
    • improve prompts/criteria accordingly
  • Use low temperature for evaluation:
    • more deterministic scoring (temperature ~0.1–0.2 mentioned)

Warning: don’t over-optimize for the proxy

If you optimize the system purely to maximize LLM-judge scores, you may create a model that beats the proxy but not the real goal (human preference). Keep the proxy aligned with ground truth as much as possible.

Dimensions evaluated: task performance vs format/safety; plus factuality

Common evaluation dimensions

  • Task performance:
    • usefulness, factuality, relevance
  • Response format alignment:
    • tone/style alignment
    • safety/unsafe content presence

Factuality evaluation methodology (step-by-step)

Goal

  • Capture factuality nuance: a response can be partly correct and partly wrong.
  • Avoid marking the entire response incorrect when only a small part is wrong.

Procedure (as described)

  1. Extract a list of facts from the full text
    • Use an LLM call to convert multi-sentence output into individual fact statements
  2. Check each fact’s correctness
    • Typically binary: correct vs not correct
    • Fact-checking can involve RAG/web search/knowledge lookup
  3. Aggregate into a single factuality score
    • Optionally weight facts by importance:
      • more important facts contribute more
    • Example aggregation: weighted sum over fact correctness.

Evaluating agentic workflows (and tool-based systems)

Why agent evaluation is harder

  • Agents run iterative loops (Observe → Plan → Act) with multiple tool calls.
  • Failures can occur at many steps, so evaluation must localize where things break.

Example: tool-based agent failure analysis (Observe/Plan/Act around tools)

Tool prediction step: finding the right tool

  • Failure mode 1: Need a tool but doesn’t use it

    • Called a “punt” (model gives up)
    • Causes:
      • tool router/selector recall failure (missing the right tool)
    • Fix:
      • adjust tool router/selector to be more recall-oriented

  • Failure mode 2: Tool not selected even though it exists

    • Fix:
      • revisit tool-calling training (SFT patterns)
      • prompt tuning/rubrics teaching when to use tools

Tool prediction step: tool name errors

  • Failure mode 3: Tool hallucination
    • Model calls a function that doesn’t exist
    • Causes:
      • model too weak/un-grounded
      • poor tool API descriptions / naming
      • unclear top-level instructions on using provided tools
    • Fixes:
      • upgrade model for better grounding
      • improve tool API naming/arguments/docstrings
      • tighten/clarify tool-use instructions

Tool prediction step: correct tool but wrong arguments

  • Failure mode 4: Argument mismatch
    • Example: wrong location coordinates because location wasn’t provided
    • Remedies:
      • ensure required context is included in prompt
      • add a location-finder tool with proper permissions
      • retrain/rewrite API so argument format is clear

Tool call execution step (tool backend problems)

  • Failure mode 5: Tool returns incorrect/vague output

    • Example: code bug returns errors
    • Fix:
      • software engineering fix for tool implementation

  • Failure mode 6: Tool returns no response

    • Especially bad for action tools (model might assume success)
    • Fix:
      • always return meaningful structured output
      • if no results, return explicit empty structured response (e.g., empty JSON) rather than “nothing”

Post-tool synthesis step: interpreting tool outputs

  • Failure mode 7: Model fails to synthesize or references wrong info
    • Causes:
      • output too large/noisy for the model to find relevant parts
      • tool backend returns data in an unhelpful format
    • Fix:
      • return structured, model-friendly objects with clear fields
      • trim/format tool outputs to highlight what matters

Debugging strategy emphasized

  • Categorize failures into groups (modeling/tool routing/tool backend/synthesis), then fix systematically.

Benchmarks for LLM evaluation (categories + examples)

Benchmark category 1: Knowledge-based (fact retrieval/recall)

  • Example: MMLU
    • Multi-task, many subject areas
    • Standardized multiple-choice questions
    • Models choose among options (less ambiguity than free-form generation)
  • Trend highlighted:
    • prefer constrained answer formats over LLM-as-judge where possible.

Benchmark category 2: Reasoning

  • Math reasoning
    • Example: AIME
      • high-school Olympiad math
      • constrained output format (e.g., three-digit answer)
      • evaluates reasoning steps → final answer
  • Common-sense physical interaction reasoning
    • Example: (mentioned) PAIR/PAS-like physical interaction QA
      • grounded in everyday/physical world
      • fewer options than MMLU; many examples

Benchmark category 3: Coding

  • Motivation: aligns with AI assistant coding use cases; tests code reading/writing.
  • Example: SWE-bench
    • built from real GitHub repositories
    • uses before/after patches + tests
    • judged by whether proposed patches make tests pass (test-driven signal)

Benchmark category 4: Safety

  • Safety benchmarking is tricky because it depends on provider policy.
  • Example: HarmBench
    • categories: standard harmful behavior, copyright-related, contextual (text modality), multimodal/contextual (other modalities)
    • some benchmarks use classifiers where explicit matching/regex is insufficient

Benchmark category 5: Agent/tool use (agent behavior)

  • Example: TaoBench
    • tool-agent/user simulation
    • defines tools + allowed/disallowed policies in domains like airline/retail
    • uses a simulated user (another large model) for multi-turn interaction
    • success measured via rewards tied to state changes (e.g., database changes)
    • metric detail:
      • inspired by pass@k, but uses pass-hat@k-style notion:
        • emphasize probability that all K attempts succeed to support reliability

How benchmarks are “grounded in reality” and caveats

  • Benchmarks are profiling tools, not universal truth.
  • Benchmarks may be adapted into “flavors” (e.g., multilingual, tool-use variants).
  • Multi-model profiling:
    • plot performance vs cost/price
    • choose best tradeoffs along a Pareto frontier
  • Data contamination risk:
    • results depend on whether the model has seen the benchmark before
    • mitigations:
      • hash values and blocklists for tool-use to avoid leaking answers
      • use held-out/unseen test sets for math (noted)
  • Final advice:
    • benchmarks don’t automatically tell whether a model is best for your needs
    • use benchmarks plus real-world testing (e.g., chatbot arena mentioned as an alternative lens)

Speakers / sources featured (as mentioned in the lecture content)

Speakers

  • In-lecture professor/instructor (main lecturer; referenced as “Ashin” when time passes to another speaker)
  • Shervin (speaker the instructor hands off to, mentioned near the end of the first major segment)

Sources / referenced systems/papers

  • ReAct (Reason + Act: Observe/Plan/Act)
  • RAG (Retrieval-Augmented Generation)
  • Sentence-BERT (bi-encoder example)
  • Cohen’s kappa, Fleiss’ kappa, Krippendorff’s alpha (inter-rater agreement metrics)
  • Constrained guided decoding / structured output (API/provider support)
  • “LLM as a judge” paper (introduced “two years ago”; title not provided)
  • METEOR, BLEU, ROUGE (rule-based evaluation metrics)
  • AIME (math benchmark)
  • PAS (physical interaction common-sense QA acronym mentioned as described)
  • SWE-bench (coding benchmark)
  • HarmBench (safety benchmark)
  • TaoBench (tool-agent benchmark)
  • MMLU (knowledge benchmark)
  • Chatbot Arena (real-world comparison method mentioned)
  • Gemini (example model/bench usage; also “Gemini flash” referenced)

Original video