Evals
evaluations
Systematic tests of an AI system’s outputs against expected results, run automatically after every change.
01In short
Systematic tests of an AI system’s outputs against expected results, run automatically after every change.
In productionNo evals, no go-live. The pass rate is the release criterion.
02Video
slot · videoEvals in 90 secondsadd: GUIDES['evals'].video
03Build your first eval set in five steps
- Pick one task and one number.Write down what a good answer means for one process — “claim routed to the right team”, not “the AI is good”. One task, one metric.
- Collect real cases.Pull a few hundred real inputs from the process, including the ugly ones: typos, edge cases, the ones people argue about.
- Label the expected outcome.Let the people who do the work today mark the right answer. Where they disagree, you found a policy question, not an AI question.
- Automate the scoring.Exact match where you can, rules where it fits, LLM-as-a-judge with a written rubric where you must — and spot-check the judge.
- Run it on every change.New prompt, new model version, new retrieval setting: the eval runs in CI and the pass rate is the gate to production.
04Checklist
05FAQ
How many cases do we need?
Enough to cover the variety of your process. A few hundred real cases usually beat thousands of synthetic ones.
Can we use a public benchmark instead?
For shortlisting models, yes. To prove the system works in your process, no — benchmarks don’t know your claims, contracts or customers.
Who owns the eval set?
The process owner. Not the vendor, and not the AI team alone — they build it together, the owner signs off the threshold.
From the magazine
06Related terms
EvaluationEval setA fixed collection of real inputs with known good outputs, scored automatically.EvaluationLLM-as-a-judgeUsing a model to grade another model’s answers against written criteria. Scales well; needs regular human spot checks to stay honest.EvaluationRegressionWhen a change — a new prompt, a new model version — breaks something that used to work.
Where we help · Secure