Method

Five questions, in this order

The order is the point. The expensive mistakes I have seen — including my own — came from starting at question three.

Find the bottleneck

Problems choose tools. Tools do not get to choose problems.

Many of the constraints an organisation wants to solve with AI turn out not to be constraints — they are habits, missing data, or symptoms of something upstream. Ask why five times and the compliance question often turns into an integration question.

Where is a faster competitor serving the same customer better?
Which high-volume decisions cost money through delay or error?
Which two departments are measured on metrics that contradict each other?
Is this bottleneck real, or a misread of where the time goes?

Finding the real bottleneck (and the fake ones)

Choose the instrument

The first job is not to sell AI.

A language model is one option among five: do nothing, delete the step, buy what exists, write the rule, or use a model. The fourth wins more often than anyone expects — cheaper, faster, auditable, testable.

Not could we — should we? What new risk does AI introduce here?
Would a deterministic rule be cheaper and more testable?
Does this build something a competitor cannot buy next quarter?
What does a wrong output cost, and who absorbs it?

Deciding when AI is the wrong tool What an AI use case actually costs

Redesign the process

Automating a bad sequence gives you a faster bad sequence.

Most processes are sequential for historical reasons. When the executor stops being one person, most of those dependencies dissolve — but only if someone goes looking for them.

Which dependency is a requirement, and which is just history?
What would this look like if waiting time were zero?
Is this manual check a control, or a comfort?
What does the current shape cost in customers who quietly give up?

From sequential to parallel

Prove it works

Eighty-six percent correct is not a result. It is a starting point.

Measuring business return tells you whether something created value. It does not tell you whether the output is good enough to release. That needs an acceptance criterion agreed before the build, a reference set that represents reality, regression on every change, and root-cause discipline that is allowed to conclude "the source data is wrong".

What is the acceptance criterion, written down before anyone builds?
Who agreed that the reference set represents reality?
When it fails — is it the model, the input, the retrieval, or the logic?
How would we notice a silent regression after a model change?

When six of seven runs failed Generating 650 bilingual assessment items with automated screening

This is where I connect established AI-evaluation practice to an explicit business release decision — and the stage I came up through.

Make it stick

A pilot is a question. It needs an answer that kills it or scales it.

Pilot purgatory is not a technology failure. It happens when nobody wrote the exit criteria, the data foundation was treated as a cathedral, roles were replaced instead of redesigned, and the team that proved the idea had no incentive to spread it.

What kills this pilot? Write it down before the pilot starts.
Which decisions are reversible enough to delegate?
Are we retraining people into new roles, or quietly replacing them?
Whose job is it to make this work somewhere else?

Escaping POC hell

Credit where it is due

What is mine, and what is not

Stages one, two, three and five are not original to me, and I would distrust anyone claiming otherwise — they are roughly where serious practitioners have landed. The reframing I have found most useful this year came from an AWS re:Invent 2025 leadership session by Helena Yin Koeppl: that organisational silos persist because incentives are misaligned rather than because the org chart says so, and that return has to be instrumented during operation, since afterwards the AI effect can no longer be separated from everything else that moved.

Stage four is where my operating model makes the release decision explicit. The underlying practices are established, not mine: the NIST AI RMF asks for representative evaluation and deployment-context validation, Google's PAIR guidance has specified pre-agreed success criteria and thresholds since 2019, MLOps literature has promotion gates, and the ML Test Score has been a production-readiness rubric since 2016. What I contribute is narrower and organisational — leadership-level material tends to move from "is this worth doing" straight to "did it pay back", and between them sits an owner, an acceptance criterion and regression evidence. Naming that gap and giving it an owner is ordinary quality engineering applied to a probabilistic system. I spent twelve years doing exactly that for software that had to ship.

Scale note: I have applied all five stages end to end on systems I built and owned, not on a transformation programme across thousands of people. Every case page says which.