← Work

Adaptive Prompt Selection & Evaluation

A design for moving beyond static prompts: retrieve candidate prompts by context, select with a multi-armed bandit, handle cold start, and close the loop with multidimensional evaluation and reward design.

Role
Designer & author of the approach
Date
October 29, 2025
Stack
Python · LLMs · Retrieval · Multi-armed bandits · Evaluation
Writing: 1 · Why Static Prompts Fail Writing: 2 · Building the Selector Writing: 3 · Reward Engineering & Evaluation

Summary

A worked design for treating prompt choice as a decision problem rather than a constant. Instead of one hand-tuned prompt, the system retrieves candidate prompts by context, selects among them with a bandit policy, handles the cold-start case, and learns from multidimensional evaluation feedback. Documented as a three-part series.

The problem

Static prompts become bottlenecks: one prompt can’t be optimal across the range of inputs a real system sees, and every manual tweak risks regressing cases that used to work.

Why it mattered

For an LLM feature that must stay accurate and consistent across varied inputs, the prompt is a high-leverage variable. Choosing it adaptively — and measuring the result — is the difference between guessing and improving.

Constraints

  • Selection has to work from the first request (cold start), before feedback exists.
  • Feedback is often delayed and multidimensional — quality isn’t a single scalar.
  • The policy must explore without destabilizing live behavior.

Role

I designed the approach and authored the series covering the problem, the selector, and the evaluation/reward loop.

Architecture

  1. Retrieval — narrow the full prompt library to context-relevant candidates.
  2. Selection — a multi-armed bandit balances exploiting known-good prompts against exploring alternatives; cold start is handled with sensible priors.
  3. Evaluation & reward — multidimensional scoring feeds a reward signal so the system learns what matters, not just what’s easy to measure.

Key decisions

  • Bandit over static A/B — continuous adaptation instead of discrete, manual experiments.
  • Retrieve before you select — keep the decision space small and relevant.
  • Design the reward explicitly — an underspecified reward optimizes the wrong thing.

Tradeoffs

  • Exploration costs some short-term quality to gain long-term adaptation.
  • Multidimensional evaluation is more work to build than a single thumbs-up metric, but a single metric hides the failures that matter.

Failure modes considered

  • Reward misspecification → the system confidently learns the wrong behavior.
  • Delayed feedback → credit assignment to the right prompt/choice becomes harder.

Outcome

A coherent, published design others can apply, spanning static-prompt limitations → contextual retrieval → bandit selection → cold start → multidimensional evaluation → reward adaptation.

In practice: the approach ran in production, where a lightweight grading signal fed back into which prompts the system used — closing the loop the series describes rather than staying on paper.

What I learned

Most of the reliability in an LLM feature lives in the loop around the model — selection and evaluation — not in any single prompt.

What I’d change next

Add a concrete, reproducible reference implementation and an evaluation harness others can run.