How many test cases does an LLM eval suite need?

Fifty to two hundred for most production features. That's enough to catch real regressions and small enough that somebody actually maintains it, which matters more than coverage — an unmaintained suite of two thousand cases gets ignored within a quarter. Source them from real user inputs rather than generating them, and include cases that have no correct answer.

Can you use an LLM to grade another LLM's output?

For narrow, factual questions, yes — "is this claim supported by this passage" is reliable enough to gate on. For broad quality judgements it isn't: a 1–10 helpfulness score drifts with the grading model's own version and can't be compared across runs. Use model grading for direction, and deterministic scoring for anything that blocks a deploy.

When should evals be written — before or after the feature?

Alongside. Before means guessing at failure modes you haven't met; after means the feature ships with no baseline, so the first regression is undetectable. In practice we write the first twenty cases while building, and grow to the full set during the week before release using whatever the build surfaced.

What do you do when the eval suite starts failing after launch?

Establish whether the system changed or the world did. If nothing was deployed, it's usually a provider-side model update or data drift — and that's exactly the case the suite exists to catch, since nothing in your CI would have gone red. Then either adapt the prompts and retrieval, or pin the model version and migrate deliberately. On a retainer this is our job; off one, it's the thing nobody notices for months.

LLM Evals for Production Applications

A production LLM eval suite needs four parts: a graded test set of 50–200 real cases, scoring appropriate to the task rather than one universal method, a regression gate in continuous integration that blocks deploys on a score drop, and continuous sampling of live traffic to catch drift the fixed test set cannot see. It is typically three to five days of work, and it is what makes an AI feature safe to change after launch.

Nobody asks for evals. Clients ask for the feature, and evals sound like the tests you write when there's spare time, which there never is.

Then six weeks after launch somebody adjusts a prompt to fix one complaint, something unrelated gets worse, and there is no way to tell. The team stops changing the system. It's working, nobody dares touch it, and it slowly becomes a fossil in the middle of the product.

This is what we actually build to prevent that. It's smaller than most eval articles suggest, deliberately.

Fifty to two hundred cases, and no more

Advice on eval set size tends toward "as many as possible", which produces a suite nobody maintains and everybody eventually ignores. We aim for fifty to two hundred cases, because that range is large enough to catch real regressions and small enough that somebody will actually update it when the product changes.

Where they come from matters more than how many there are:

Real user inputs, from support tickets, search logs or the pilot. Synthetic cases test the system you imagined rather than the one people use.
The awkward ones deliberately included. Ambiguous phrasing, typos, two questions in one message, the wrong language. These are where regressions surface first.
Cases with no correct answer. Questions your corpus genuinely can't answer. A confident response to one of these is a *failure*, and it must be tested for explicitly or the system will learn to always say something.
Adversarial cases, if the model reads untrusted input: a document containing instructions, a query trying to reach another tenant's data.

Scoring: what we trust and what we don't

There is no universal scoring method, and picking the wrong one produces a number that moves without meaning — which is worse than no number, because people act on it.

Task	How we score it	Trust
Structured extraction	Field-by-field comparison against expected output	High. Deterministic and cheap.
Classification	Accuracy and per-class recall against labels	High.
Retrieval	recall@k and mean reciprocal rank against known-good passages	High. Scored before generation, always.
Grounded answering	Faithfulness — is every claim supported by a retrieved passage	Medium. Model-graded, but a narrow question.
Open-ended prose	Model-graded against a written rubric	Low-medium. Useful for direction, not for a pass/fail gate.
"Overall quality", 1–10	—	None. Don't. The number drifts with the grading model and means nothing across runs.

Match the method to the task. Most real suites use two or three of these together.

The general rule: the narrower the question you ask a grading model, the more you can trust the answer. "Is this claim supported by this passage, yes or no" is reliable. "Rate the helpfulness of this response from one to ten" is a random number generator with good manners.

Score retrieval separately, always

For any retrieval-backed feature this is the single most valuable structural decision in the suite. Score whether the right material was found, then separately score what the model did with it.

Without the split, a bad answer is undiagnosable — you can't tell whether retrieval missed or generation failed, so you tune the prompt because the prompt is visible. That's how afternoons disappear into rewording an instruction that was already fine.

A regression gate that actually blocks

The suite runs on every pull request that touches a prompt, a retrieval parameter, a model version or the context assembly. It fails the build on a meaningful drop.

# Fails the build when accuracy drops more than 2 points
# below the score recorded on main.
eval:
  dataset: evals/cases.jsonl
  gates:
    - metric: extraction_field_accuracy
      min_absolute: 0.90
      max_regression: 0.02
    - metric: retrieval_recall_at_5
      min_absolute: 0.85
      max_regression: 0.03
    - metric: refuses_when_unanswerable
      min_absolute: 1.00   # no tolerance on this one

The threshold is a band, not a fixed floor — model-graded scores have run-to-run noise, and a gate that fires on noise gets disabled within a fortnight.

Two details that decide whether the gate survives. It must be fast — a suite taking twenty minutes gets skipped. And it must be cheap: running two hundred cases against a frontier model on every commit is a real bill, so we usually run a fast subset per commit and the full set nightly.

The fixed test set ages. Sample production.

A test set frozen at launch measures how well the system handles last quarter's questions. Real inputs drift as users learn what the feature does and as your data changes underneath it.

Sample a slice of live traffic — a percentage, or every request that trips a heuristic — and score it with the same graders.
Watch the cheap signals. How often users rephrase and ask again, abandon a generation, or click through to a cited source. Rephrasing is the most useful negative signal you'll get for free.
Promote failures into the test set. This is the loop that matters: production surfaces a new failure mode, it becomes a permanent case, and it can never silently return.

What guardrails add, and what they don't

Evals tell you how often the system is wrong. Guardrails decide what happens when it is. They're complementary and frequently confused.

Schema validation on structured output, with a defined path — one retry with the errors fed back, then a human queue. Never a silent partial record.
Grounding checks on cited answers: if a claim can't be traced to a retrieved passage, don't show it.
Per-user and per-tenant cost caps, so one retry loop can't produce a five-figure invoice.
Least-privilege tools for anything that acts, scoped to the calling user's own permissions.
A refusal path, tested, so the system can say it doesn't know.

None of these fix a bad model output. They contain it, which is the achievable goal. The integration patterns post covers where each one belongs architecturally.

Three to five days

That's what this costs to build alongside a feature, and it's the difference between a system you can improve and one nobody is allowed to touch. It's part of every AI project we take on rather than a line item to be negotiated away — see how we scope it.

The tooling question, incidentally, matters less than any of this. We've built these on hosted platforms and on a few hundred lines of TypeScript writing JSON to a file. The suite's value is in the cases and the honesty of the scoring, not the dashboard around them.

The evals we build before shipping an LLM feature