How to Plan an AI Evaluation: An LLM Eval Workflow That Catches Real Failures

Most teams ship an LLM feature, watch it pass a demo, and call that "tested." Then a support reply hallucinates a refund policy, a classifier silently mislabels half the long inputs, and nobody can say whether the model got worse because there was nothing to compare against. A useful eval pins down three things before you ship: the task, a representative test set, and clear metrics. A handful of cherry-picked examples gives false confidence, so the set has to cover the real distribution and the edge cases that actually break things in production.

This guide walks through planning an AI evaluation as a workflow you can run in an afternoon, not a research project. If you want to skip straight to generating cases, the AI Eval Planner turns a feature description, risk list, and user path into a first draft of test cases and pass criteria.

Start by defining the task, not the model

The first mistake is grading the model when you should be grading the feature. "Is GPT good?" is unanswerable. "Does the summarizer keep every dollar amount from the source invoice?" is a test.

Write the task as a contract: given this input, the output must do X and must never do Y. For a summarization feature, the contract might be: produce a 3-sentence summary that contains every named party and every dated obligation, never invents a clause, and stays under 60 words. For a support classifier: route the message to one of seven intents, abstain when confidence is low rather than guessing, and never label a refund request as a general question.

Once the task is a contract, you know exactly what the eval has to measure. Every fuzzy goal like "be helpful" should decompose into something a second reviewer could score the same way you did. If two people on your team would disagree about whether an output passed, the criterion is too vague to ship.

Build a test set that mirrors reality

The test set is where evals live or die. A common failure is assembling 15 friendly examples that the model already handles, watching them all pass, and declaring victory. That set tells you nothing because it doesn't contain the cases that fail.

A representative set has three layers. The first is the happy path: typical inputs at typical length, sampled from real traffic if you have it. The second is the distribution tail: the longest documents, the shortest fragments, the languages you didn't design for, the malformed pasted text. The third is risk-driven cases, one per concrete failure mode you can name. List those failures explicitly: hallucinated policy, privacy leakage, schema drift, unsafe advice, wrong language. Each one becomes a case with an expected behavior.

Size matters less than coverage, but there is a floor. A 12-case eval will swing wildly on a single flipped result, and you will overfit to it without noticing, tuning the prompt until those exact 12 pass while the real distribution rots. Aim for enough cases that no single example moves the headline number by more than a couple of points, and keep a held-out slice you never tune against. When the tuned set and the held-out set start disagreeing, you've been overfitting.

Choose metrics that map to the contract

Pick metrics that match what the task contract promised, and write down the threshold before you run anything.

Accuracy / correctness: for classification, this is straightforward — labels match ground truth. For generation, it's per-claim checks: did every required fact survive?
Faithfulness: does the output stay grounded in the source and avoid inventing? This is the metric that catches hallucination, and it's the one demos never test.
Latency: p50 and p95 response time, because a correct answer that takes 14 seconds fails a chat feature.
Cost: tokens per request times your price. A faithful answer that costs 40 cents may still be the wrong design. The LLM Pricing Calculator is handy for turning token counts into a per-1,000-request budget while you're deciding which model tier the eval should target.

Set a pass bar per metric ahead of time. "Faithfulness ≥ 98% on the risk set, accuracy ≥ 90% on the held-out set, p95 under 3 seconds, under 2 cents per request." Deciding the bar after you see the scores is how teams talk themselves into shipping something broken.

Split automated grading from human grading

Not every metric needs a person, and not every metric can be automated. Decide the split per criterion.

Automate the objective ones: label matching, schema validation, regex checks for forbidden strings, latency, and cost are deterministic and should run on every change. Faithfulness and tone usually need either a human reviewer or an LLM-as-judge, and a model judge needs its own calibration — spot-check 30 of its verdicts against a person before you trust it at scale, because a miscalibrated judge gives you confident garbage.

The practical rule: automate everything you can express as a rule, sample everything else with humans, and treat the LLM judge as a force multiplier on human review, never a replacement for it.

A worked example: a classification feature

Say you're shipping a feature that classifies inbound support messages into seven intents. Here's the eval plan.

Task contract: assign exactly one of seven intents, or abstain when no intent fits; never route a billing dispute to "general."

Test set: 200 real messages stratified across all seven intents (so rare intents aren't drowned out), plus 30 adversarial cases — typos, mixed languages, two intents in one message, empty bodies. Hold out 50 of the 200 as the set you never tune against.

Metrics and bars: macro-F1 ≥ 0.85 (macro so a rare intent can't hide behind a common one), abstention precision ≥ 0.95 on the held-out slice, p95 latency under 1.2s, cost under 0.3 cents per message.

Grading split: label match and latency are fully automated; the 30 adversarial cases get human review because "correct intent for an ambiguous message" is a judgment call.

You run it, get macro-F1 0.88 but billing-vs-general confusion at 0.71 recall — a real defect the demo never surfaced, because the demo used clean billing messages. That's the entire point of the eval.

My own habit

I keep one rule for myself: I never let the eval set be smaller than my confidence in it. The first time I tuned a prompt against eight cases, I got all eight green and shipped something that fell over on its first real day. Now I write the failure modes down before I write a single case, generate a draft suite, and split it into a tuning half and a held-out half on day one. When the two halves disagree, I stop tuning — that gap is the model telling me I've been teaching to the test.

Putting it together

A good eval plan is mostly upfront thinking: a task written as a contract, a test set that covers the happy path and the tail and the named risks, metrics tied to that contract with thresholds set in advance, and a clear line between what a script grades and what a person grades. Generate the first draft of cases with the AI Eval Planner, then keep the held-out slice honest so the number you report is one you can actually trust.

Made by Toolora · Updated 2026-06-13