Video summary

Automated Prompt Optimization - Mikhail Sveshnikov

Main summary

Key takeaways

Educational

Main Ideas / Concepts Covered

  • Evidently (open-source) is presented as an AI evaluation framework that started with tabular/ML-style evaluation and expanded to LLM evaluation features, including “LLM as a judge” (LLM-driven classification of text against criteria).
  • A core problem: Writing good prompts for LLM judges is hard:
    • Requires domain expertise
    • Is time-consuming
    • Small prompt changes can cause large performance differences
    • It can be difficult to know when a prompt is truly optimal
  • Prompt optimization is proposed to improve LLM judge prompts using labeled evaluation data:
    • Instead of manually crafting instructions/rules, the system learns prompt improvements from examples (labels + failure cases).

  • The talk surveys four families of prompt optimization approaches:

    1. Search-based (test many prompt variants, pick the best)
    2. Model-internals (use gradients/embeddings/token distributions; requires access to model weights and inference internals)
    3. Self-improvement (ask an LLM to rewrite/improve the prompt)
    4. Example-based (optimize demonstrations/few-shot examples rather than instructions)

  • The speaker’s decision: build a solution that combines strengths, tailored for real-world usage where users may not have model internals:

    • Works with limited label data
    • Produces interpretable results
    • Integrates with Evidently’s LLM judge workflow
    • Is practical/easy to use

Methodology / Algorithmic Approach (Evidently Prompt Optimizer)

A) High-level workflow (architecture)

The feature is exposed via an API entry point conceptually called PromptOptimizer, using three components:

  • Strategy

    • Takes accumulated prompt logs (initial prompt, previous prompts, scores, executor results)
    • Generates a new prompt to try

  • Executor

    • Runs the current prompt to produce results
    • Typically implemented as a function (e.g., “evaluate the LLM judge on a dataset”)
    • Can be customized

  • Scorer

    • Takes executor outputs and computes numeric success scores
    • Example: accuracy (or a custom scoring function)

B) Core optimization loop

For each optimization run:

  • Start with an initial prompt
  • Repeat in a loop:
    • Use executor to run the prompt and obtain results
    • Use scorer to score those results
    • Store everything as a prompt log (prompt + results + scores)
    • Decide whether to stop:
      • If early stopping conditions are met → return best prompt(s)
      • Otherwise → call strategy to generate the next prompt and continue

C) Two built-in strategies

1. Simple strategy

  • Asks the LLM to “improve my prompt” using the user prompt and any provided instructions.
  • Useful especially when label data is unavailable (per the talk).

2. Feedback strategy (main contribution; used in demos)

Steps:

  1. Execute the current prompt on training data
  2. Identify mistakes where the judge/classifier output is wrong
  3. Collect mistakes and embed them into a meta prompt
  4. Ask the LLM to generate an improved prompt using those mistakes

Additional input signals used per mistake:

  • Text input
  • Target label
  • Incorrect prediction
  • Optional reasoning:
    • If using Evidently’s LLM judge, a flag can include the judge’s reasoning
    • If human-labeled, humans can provide “why” explanations so the LLM can extract patterns

“Secret sauce” elements emphasized:

  • Generalize examples to avoid overfitting
    • Without this, the optimizer may overfit by copying specific failing examples into the new prompt.
    • The feedback meta prompt includes instructions intended to generalize.
  • Prevent overfitting using train/validation/test split
    • Default split: 40% train / 40% validation / 20% test
    • Process:
      • Optimize on train (only failures from train are used)
      • Select the best prompt using validation scores
      • Report final performance on a held-out test set never seen by the optimizer

Early stopping and controls (configurable parameters)

  • Max iterations
    • Maximum number of prompt-improvement loops
  • Minimal score gain
    • Stop if improvement between iterations is below a threshold
  • Target score
    • Stop if reaching a desired performance level (e.g., 100% or 90%)
  • Multiple starts (repetitions)
    • Run optimization multiple times because LLM outputs are stochastic
    • Select the best prompt across runs
    • The talk compares it conceptually to bagging-like effects, but emphasizes selecting the best prompt rather than averaging prompts

D) Performance features in implementation

  • Parallel execution of optimization runs / prompt evaluations
  • Async + throttling for efficient LLM API usage
    • Example mentioned: staying under request quota (e.g., ~500 requests/min for OpenAI)
  • Caching where possible
  • Full audit logs
    • Track every iteration: prompts, scores, timing, token usage, etc.
    • Supports debugging and transparency

E) Simple prompt-optimization demo concept (from live coding)

  • A classification task is set up with labels mapped to 0/1.
  • A deliberately bad prompt is used first (“Classify this text. Return only zero or one.”).
  • The system evaluates accuracy on labeled examples.
  • A meta prompt is created:
    • Provides the observed mistakes
    • Asks the LLM to improve the original prompt
    • Requests the response be wrapped in a <new_prompt> tag for parsing
  • After applying the improved prompt, the demo shows near-perfect performance for that simplified task.

Use Cases / Practical Guidance Mentioned

  • Binary or multi-class evaluation works well when the judge outputs map to discrete labels (e.g., relevant/not relevant, helpful/not helpful).
  • If you can define evaluation criteria that translate into a label-based correctness signal, you can use the prompt optimizer even without full LLM monitoring.
  • Prompt optimization can be used in two ways:
    • Optimize the LLM judge criteria prompt/wrapper (when the judge is the thing being evaluated/used)
    • Use an LLM judge as a scorer to optimize an agent prompt:
      • The agent generates responses
      • The judge evaluates them
      • The judge score becomes the optimization target

Q&A Takeaways (Key Answers Summarized)

  • Long/structured prompts
    • Likely works as long as content fits in the model’s context window.
    • Overly complex structures may break; suggestion: optimize only the meaningful criteria portion while keeping structural wrappers stable.
  • Overfitting concern
    • Addressed via train/validation/test splitting and a generalization instruction in the meta prompt.
  • DSPy / TextGrad comparison
    • DSPy research wasn’t used in their approach because it pushes users into the DSPy ecosystem (more rewriting and LLM-call plumbing).
    • Evidently is framed as easier to plug into an existing prompt/agent workflow.
  • Optimizer vs evaluator model
    • Recommendation: use a smarter model for prompt optimization/judging.
    • Examples often use the same model family, but the design allows specifying models for the judge; separating models is possible via configuration.
  • Did you optimize the judge or the agent?
    • Clarified: depends on how executor/scorer are defined.
    • You can optimize LLM judge criteria, or have a judge score an agent output to optimize the agent prompt.

Speakers / Sources Featured

  • Mikhail Sveshnikov (main speaker; presented prompt optimization approach and Evidently implementation)
  • Emily (mentioned as being present in live chat; referenced from an earlier Evidently demo/story)
  • Data Talks Club (event organizer/community; also mentioned as hosting/promoting the webinar)
  • ChatGPT / OpenAI (referenced as tools/models in examples and discussion; not as a speaker)
  • Evidently library / Evidently (open-source) (primary software/source discussed and demonstrated)
  • LinkedIn profile (Mikhail’s; referenced at the end)

Original video