The Art of Validation

← back to the series map

Here's the most expensive lesson I've learned building with agents:

Verification is not validation

Verification asks: did we build the thing right? Validation asks: is it the right thing? They feel like the same activity. They are not, and the difference is where teams building with agents quietly die.

Here's the physics of it: an agent can generate verification endlessly — more tests, more checks, more green — because verification is something a system can produce about itself. Validation it structurally cannot supply, because the answer lives outside the system: in the person the work is for, in the promise it was supposed to keep. Nobody ever drifts into validation. Everyone drifts into verification, because it's the thing that's always on tap.

The scar

We finished a system that watches my inbox and files real sales conversations into our CRM. By the time we called it done it had 11,207 passing tests and had survived eight independent adversarial reviews. Everything green, eight times over. Then we ran it against my actual inbox for the first time — and it wanted to file a vendor's support ticket as a sales lead.

Nothing in the green was lying. Every test proved the machinery did exactly what we'd imagined. That's the trap: all 11,207 were verification. Nobody had asked what the system would decide about reality — the validation question — until reality answered it for us.

The lesson

Validation is owning the finish line. Name the promise the work makes — what needs to be true, for whom, with what force in the real world — and hold it constant while everything underneath is allowed to change. Then give the agent a way to verify that. An agent left to define its own finish line will run to it and declare victory, honestly and uselessly.

How to run it tomorrow

1. Make it name the promise and the consumers. Before accepting done: "List everyone and everything that consumes this work, and what each needs to be true." If the list surprises you, the checks were aimed wrong.
2. Ask what would make the green a lie. "All your checks pass. What could still be wrong that none of them would catch?" The honest answer is where validation actually starts.
3. Feed it reality — and read your users as instruments. Run the real path on real data and have it list every disagreement, not a score. And when a real person hits something, hear it as evidence about the system, not a ticket: the complaint is the highest-fidelity validation signal you own.
4. Make failures cheap and enumerated — and the tests honest. Move discovery to the cheapest surface that still tells the truth; never pay production prices for an error a cheap pass could have enumerated. Stress the range — easy to crazy hard — and never accept a synthetic proof where a live measurement was demanded.
5. Walk it as the user. Real environment, cold, before "done." Checks catch lies in the code; the walk catches truths that read wrong.

The agent brings endless verification. You bring the validation. That division of labor is the art.