Topics / Evaluation
Evaluation
5 pieces touching Evaluation, newest first.
- The Agent Said the Tests Passed. No Tests Ran.
An agent graded its own CI run as a pass when no pipeline had run. The fix was an evidence ledger the agent has no write path to, and a guard that reads the ledger before it believes the report.
- Ground-Truthing Your LLM Judge: Does Your Eval Actually Track Reality?
LLM-as-judge is the default eval now, but a judge is just another unreliable model. Here is how to check whether its verdicts correlate with real outcomes, and catch the leniency bias most judges have.
- Reward Engineering and Evaluation
This is the third post in our series on probabilistic prompt pipelines. In our first post, we explored why static prompts become bottlenecks. The second…
- Building the Selector: Retrieval, Bandits, and Cold Start Solutions
This is the second post in our series on probabilistic prompt pipelines. In the first post, we explored why static prompts become bottlenecks and saw a…
- Why Static Prompts Fail and How Probabilistic Selection Fixes It
This is the first post in a four-part series on building production-ready probabilistic prompt pipelines. By the end of this post, you'll understand why…