Evaluating LLM Outputs: Evals, Graders and Judges
Why assertEquals cannot test an LLM feature, how a golden dataset plus graders catches regressions, and where LLM-as-judge, statistics and CI gates fit.
An LLM (large language model) feature cannot be tested with assertEquals: the same prompt can return different text on every run, and many different answers are correct. An eval replaces the exact-match assertion with a fixed dataset of real inputs plus graders that score properties of each output (valid JSON, the right label, every claim supported by the source), and runs it on every prompt or model change, comparing the candidate against the current baseline. Without one, each "improvement" you make by eye quietly breaks cases you are not looking at.
Context
Machine learning has always measured models against held-out test sets, and natural language processing grew automatic metrics for text: BLEU (bilingual evaluation understudy, 2002) for translation and ROUGE (recall-oriented understudy for gisting evaluation, 2004) for summaries, both counting overlapping words with a reference answer. Large models broke those habits from two sides. Their answers are open-ended, so word overlap says little about quality, and teams now ship them through prompts rather than training runs, so the people changing behaviour are application developers who were never handed a test set. Academic benchmarks such as HELM (2022) and MT-Bench (2023) measure models in general; the practice that spread from 2023 onwards is the product eval: a small, task-specific dataset owned by the team that owns the feature.
You have already met the problem if you changed a system prompt to fix one complaint and a different kind of request started failing a week later. The smallest useful eval is a list of inputs with a check on each output:
const cases = [
{input: 'Refund for order 1182, it arrived broken', label: 'refund'},
{input: 'Where is my package?', label: 'shipping'},
{input: 'Cancel and also change my address', label: 'multi'},
]
let passed = 0
for (const c of cases) {
const out = await classifyTicket(c.input) // your LLM feature
if (out.label === c.label) passed++
}
console.log(`${passed}/${cases.length}`) // 2/3 — now you can compare- Eval
- A repeatable run of a feature over a fixed dataset, with graders that turn each output into a score. The LLM equivalent of a test suite.
- Golden dataset
- The curated set of inputs, and where possible expected outputs or properties, that every candidate is run against.
- Grader
- Code or a model that scores one output: a regex, a JSON-schema check, a similarity score, an LLM judge or a human.
- Baseline
- The prompt and model currently in production. Candidates are judged by their difference from it, not by an absolute score.
- LLM-as-judge
- Using a language model with a written rubric to grade another model’s output where code cannot.
- Groundedness
- Whether every claim in an answer is supported by the context it was given, the key property for retrieval-augmented answers.
Why it matters
Prompt and model changes are code changes with no compiler and no type checker. A reworded instruction, a new few-shot example, a provider updating the model behind an alias or a switch to a cheaper model can each shift behaviour on inputs nobody re-read. Teams without evals iterate by reading five outputs and deciding it "looks better", which optimises for the five cases on screen and regresses the rest. With an eval, the same change becomes a diff: 3 cases fixed, 7 broken, cost down 40%, and the decision is no longer a matter of taste.
The eval loop
An eval is not a one-off benchmark; it is a loop that turns production failures into permanent regression cases. Every candidate (a new prompt, model or retrieval setting) runs over the same dataset as the baseline, the graders score both, and the report shows where they differ. Only a candidate that wins without breaking protected cases moves on to a canary in production, whose traces feed the next round of the dataset.
- 1Freeze the dataset version and the baseline's outputs (or re-run the baseline in the same job), so the only variable is the candidate.
- 2Run the candidate over every case, ideally several times per case at the production temperature, recording output, tokens, latency and cost.
- 3Score with graders in order of cost: deterministic checks on everything, the LLM judge only where code cannot decide, humans on a sample and on disagreements.
- 4Report per case and per slice (language, customer tier, input length), not just one average, and list the cases that flipped from pass to fail.
- 5Gate: block the change if protected cases regress or the score drops beyond noise; otherwise ship behind a canary and watch the online signals.
Graders, cheapest first
A grader turns one output into a number or a pass/fail. Most of the value comes from the boring ones: many LLM features return structured data or a choice from a fixed set, and those can be checked exactly. Open-ended text needs a reference to compare against or a judge, and every step up the table costs more and agrees less with itself.
| Grader | Checks | Good for | Weak at |
|---|---|---|---|
| Code checks | Exact label, regex, JSON schema, field values, length, banned words | Classification, extraction, format, safety rules | Anything with many valid phrasings |
| Reference | Overlap (BLEU, ROUGE) or embedding similarity to a written answer | Short factual answers, translation | Open text: a correct answer worded differently scores low |
| LLM judge | A model scores against a rubric or picks the better of two answers | Helpfulness, tone, groundedness, instruction following | Biased and noisy; must be validated against humans |
| Human | Expert or trained raters with the same rubric | Ground truth, calibrating the judge, high-stakes cases | Slow, expensive, raters disagree too |
type Case = {id: string; input: string; expect: {label: string}}
type Score = {id: string; pass: boolean; why: string; costUsd: number}
async function grade(c: Case, run: Feature): Promise<Score> {
const res = await run(c.input) // {text, costUsd}
let parsed: {label?: string}
try {
parsed = JSON.parse(res.text) // check 1: valid JSON
} catch {
return {id: c.id, pass: false, why: 'invalid json', costUsd: res.costUsd}
}
const ok = parsed.label === c.expect.label // check 2: right label
return {id: c.id, pass: ok, why: ok ? '' : `got ${parsed.label}`,
costUsd: res.costUsd}
}
export async function compare(cases: Case[], base: Feature, cand: Feature) {
const a = await Promise.all(cases.map(c => grade(c, base)))
const b = await Promise.all(cases.map(c => grade(c, cand)))
const broke = b.filter((s, i) => a[i].pass && !s.pass).map(s => s.id)
const fixed = b.filter((s, i) => !a[i].pass && s.pass).map(s => s.id)
return {broke, fixed,
base: a.filter(s => s.pass).length, cand: b.filter(s => s.pass).length}
}LLM-as-judge, and how it lies
When the property is "is this answer supported by the retrieved documents" or "does it follow the style guide", a second model with a rubric is the only scalable grader. It works best with narrow, binary questions ("does the answer contain a claim not in the context? yes/no, quote it") rather than a 1-10 score, and with pairwise comparison ("which of these two is better?") when you are choosing between candidates. Zheng et al. (2023) documented the biases to design around: judges prefer whichever answer comes first (position bias), prefer longer answers (verbosity bias), and tend to favour output from their own model family (self-preference). Swap the order and average both runs, keep rubrics concrete, and before trusting a judge, measure how often it agrees with human labels on a few hundred cases.
You are grading an answer for groundedness.
CONTEXT:
{{retrieved_passages}}
ANSWER:
{{model_answer}}
List every factual claim in ANSWER. For each, quote the CONTEXT
sentence that supports it, or write UNSUPPORTED.
Then output JSON only:
{"unsupported_claims": [...], "grounded": true | false}
"grounded" is true only if unsupported_claims is empty.
Do not judge style, length or helpfulness.Building the dataset and reading the numbers
A dataset of invented examples tests what the author imagined. The useful cases come from production: real inputs sampled across segments, every input a user flagged, every incident, and the edge cases support keeps seeing (empty input, another language, two requests in one message, prompt-injection attempts). Label them carefully, ideally with two people and a written guideline, because a wrong expected answer makes a correct model look broken. Version the dataset like code, and keep a held-out slice you do not look at while tuning the prompt, or you will overfit to it the same way a model overfits a training set.
Then treat scores as measurements with noise. On 50 cases, one flipped answer moves accuracy by two points; at a non-zero temperature, the same candidate can score differently on two runs. Run each case several times, report a confidence interval or at least the run-to-run spread, and only call a difference real when it exceeds that spread. Always compare against the baseline in the same run rather than against last month's number, because the model behind an alias, the dataset and the graders all drift.
| Feature | What to measure |
|---|---|
| Classification | Precision and recall per label, confusion matrix; accuracy hides a label that is never predicted |
| Extraction | Per-field exact match, schema validity, rate of hallucinated fields |
| RAG answers | Retrieval recall (was the right passage retrieved?), groundedness, answer correctness, refusal when the answer is absent |
| Agents | Task success, number of steps and tool calls, wrong-tool rate, cost and wall time per task |
| Every feature | Cost per request, p95 latency, output length; a cheaper model that is equally good is a win |
Pitfalls
- Judging by eye on a handful of examples
Reading five outputs optimises the prompt for those five. LLM changes have wide, non-local effects, so the regressions land on inputs you did not look at. Even 50 real cases with simple checks catch far more than careful reading of a few.
- Trusting an uncalibrated LLM judge
A judge has position, length and self-preference biases and can be confidently wrong about the rubric. If you never measured its agreement with human labels, you do not know whether a score change is the candidate or the judge. Calibrate on a labelled sample, and re-check when you change the judge model.
- Reading noise as progress
A two-point gain on 60 cases at temperature 0.7 is usually nothing. Without repeated runs and a sense of the spread, teams ship changes that did not help and revert ones that did. Measure the variance of the baseline against itself first; that is your noise floor.
- Overfitting the eval
Once the score is the target, prompts grow special cases for dataset examples and the number climbs while users see no change (Goodhart's law). Keep a held-out set, keep adding fresh production cases, and treat a sudden jump in score with suspicion rather than celebration.
- Ignoring cost and latency
A candidate that is one point better and three times slower or pricier is often a worse product. Record tokens, cost and latency per case in the same report, so the trade-off is visible when the decision is made rather than on the next invoice.
Interview questions
Q1Why can’t you unit test an LLM feature the normal way?
Because outputs are non-deterministic and many different outputs are correct, so asserting an exact string either fails constantly or tests nothing. You still test, but you assert properties instead of strings: valid JSON, the right label, no unsupported claims, length limits. And you assert them over a dataset with a pass rate compared against the baseline, not on one example.
Q2Walk me through setting up evals for a new support-ticket classifier.
I would sample a few hundred real tickets across categories and languages, have two people label them with a written guideline, and resolve disagreements. Graders are code: valid JSON, label exact match, and per-label precision and recall. A runner executes baseline and candidate on every case a few times, reports the confusion matrix, the cases that flipped and the cost, and fails CI if protected cases regress. Misrouted tickets from production get added every week.
Q3When would you use an LLM as a judge, and how do you make it trustworthy?
When the property cannot be checked by code, such as groundedness, tone or instruction following. I keep the rubric narrow and binary, ask the judge to quote evidence, use pairwise comparison with the order swapped to cancel position bias, and validate it against human labels before relying on it. If agreement is low, the judge is measuring itself, not the candidate.
Q4What happens when the provider updates the model behind the alias you use?
Behaviour can change without any change in your code, so prompts tuned to the old model may regress. Pin a dated model version where the provider offers one, run the eval suite against the new version before switching, and keep online monitors on live traffic so a silent change shows up as a drop in groundedness or a spike in fallbacks rather than as user complaints.
Q5How do you evaluate a RAG pipeline?
In two parts, because retrieval and generation fail differently. For retrieval: did the top results contain the passage that answers the question, measured as recall at k on labelled questions. For generation: is every claim grounded in the retrieved context, is the answer correct, and does it refuse when the answer is not there. Separating them tells you whether to fix the index or the prompt.
Q6Your eval score went from 81% to 84%. Do you ship?
Not on that number alone. I check how many cases that is, how much the baseline varies between runs, whether any protected or high-stakes cases broke, how cost and latency moved, and whether the gain holds on the held-out slice. If it survives all of that, it goes to a canary, and the online signals decide.
Q7How do you build the eval dataset if you have no production traffic yet?
Start with what you have: hand-written cases from the spec, examples from domain experts, and adversarial cases you can imagine, clearly marked as synthetic. Ship to a small audience early, log everything with consent, and replace synthetic cases with real ones as they arrive, because real inputs are messier than anything you invent.
- LLM features are tested by properties over a dataset, not by exact strings on one example.
- Use the cheapest grader that can decide: code checks first, an LLM judge where code cannot, humans to calibrate.
- LLM judges have position, verbosity and self-preference biases; swap orders, keep rubrics binary, measure agreement with humans.
- Build the dataset from real traffic and failures, label it carefully, version it, and keep a held-out slice.
- Compare against the baseline in the same run, know your noise floor, and track cost and latency next to quality.
- Offline evals make a change safe to try; canaries and online signals decide whether it is better.