The missing chapter — how do you know it is good enough to ship?

Read the serious material on enterprise AI adoption and you will find a consistent, sensible arc. Start from business strategy rather than technology. Find where the organisation actually loses. Ask should we and not only could we. Redesign the process instead of automating a step inside it. Instrument return during operation rather than reconstructing it afterwards, because once a change is live the AI effect and everything else that moved cannot be separated.

I agree with all of it. What I keep noticing is what sits between the second-to-last step and the last one.

You have decided the use case is worth doing. You have not yet measured whether it paid back, because it has not shipped. Someone has to decide whether the output is good enough to put in front of a customer.

That decision has an owner in traditional software. It has a name, a set of practices, and a profession behind it. In most AI programmes, it has none of those.

Why the gap exists

Partly it is unglamorous. Deciding which use cases deserve funding is strategy work; deciding whether output quality clears a bar is not the kind of thing that gets a slide.

Partly it is that AI output looks finished. A wrong answer from a language model arrives with the same fluency and confidence as a right one. There is no compiler error, no stack trace, no visibly broken layout. The failure mode of generative systems is plausibility, and plausibility defeats casual review.

And partly it is a category error. Teams reach for their existing test suite, find it green, and conclude the system works. But a unit test asserts that code does what the code was written to do. It says nothing about whether the system produces the right answer for a photograph taken under a ceiling light, or a document formatted the way that one regional office formats them. I have watched a fully green test suite coexist with six of seven real-world cases failing. Both facts were true. They were measuring different things.

What the chapter contains

The practices are not new. They are ordinary quality engineering, pointed at a probabilistic system.

An acceptance criterion, written before the build. Not “it should be accurate” — a specific, falsifiable bar: which categories of input, what rate on each, which failure modes are acceptable and which are blocking. Written down while everyone is still optimistic, because after launch the bar moves to wherever the system happens to land.

A reference set that represents reality. Real cases with known-correct answers, assembled from the actual input distribution rather than the convenient one, owned by someone, and ideally built before you know how the system performs. A reference set constructed after the failures is a justification; one built before is evidence.

Regression on every change. Prompt edits, model version bumps, retrieval changes, schema changes upstream. Any of them can move quality silently, and silence is the point — nothing announces a regression in a generative system. If your provider deprecates a model in six months, revalidation is not an unfortunate surprise, it is a scheduled cost.

Root-cause discipline, including the answer nobody wants. When it fails, the question is which component: input handling, retrieval, ranking, business logic, or the model. In my experience the model is the least frequent answer. And sometimes the correct conclusion is that the source data does not contain the answer at all — in which case failing is the right behaviour and “fixing” it would be teaching the system to invent a confident response for exactly the inputs where being wrong costs most.

Behavioural verification. Check the running system, not its configuration. I have been burned by a deployment that reported one version while executing another; the only reliable proof was invoking it and reading what actually came back.

Why it belongs at the strategy table

Because without it, three things break at once.

You cannot make a release decision, so it gets made by whoever feels confident that day. You cannot attribute return honestly, because you do not know which portion of output was usable. And you cannot delegate authority to the system safely, since the reversibility question — what may run unsupervised — depends entirely on knowing the error profile.

That last point is the one I would put in front of an executive. Every conversation about AI autonomy is really a conversation about measured reliability. If you cannot say what the system gets wrong and how often, you are not in a position to decide what it is allowed to do alone. The governance discussion and the evaluation discussion are the same discussion, and most organisations are having only the first one.

The transferable part

I spent twelve years deciding whether software was good enough to ship, in an environment where a bad release was visible to millions within minutes. The artefacts were acceptance criteria, risk based prioritisation, release quality gates and KPI frameworks. The system is probabilistic now and the tooling is different.

The question has not changed at all.