Deciding when AI is the wrong tool

The first job of anyone advising on AI is not to sell AI. It is to answer a narrower question honestly: given this constraint, which instrument fits? A language model is one option among five, and it is frequently the worst of them — slower, more expensive, less predictable, and harder to prove correct than a rule that a developer could write in an afternoon.

I treat this as a gate that closes before any model is chosen. If the modality question has not been answered, the model question is premature.

The five options, in the order I consider them

1. Do nothing. The constraint is real but the cost of solving it is higher than the cost of living with it. This is a legitimate outcome and should be written down, not silently skipped.
2. Change the process. If the step exists out of habit, deleting it beats automating it.
3. Existing software. Somebody has already solved this, and it is a configuration problem.
4. Deterministic automation. The rules are knowable and stable. This wins on cost, latency, auditability and testability — four things a language model is bad at.
5. A language model. The input is genuinely unstructured or variable, the rules cannot be enumerated, and a probabilistic answer with a known error profile is acceptable.

The instinct to jump to (5) is what makes AI programmes expensive. The discipline is to make (5) earn its place against (4).

Which instrument fits?

The same five questions I work through before a model is chosen. It will not decide for you — it shows which option the answers point at, and why the others were ruled out.

How structured is the input?

Fixed fields, predictable format Mostly structured, some variation Free text, images, documents nobody standardised

Can the rules be written down?

Yes — stable and enumerable Partly, with many exceptions No — they change faster than we can write them

What happens when an output is wrong?

Serious: money, safety, legal or reputational harm Annoying: someone has to notice and redo it Cheap: caught downstream, low cost to fix

How often does this happen?

Rarely — a handful of times a month Regularly — daily Constantly — hundreds or thousands of times

How much human judgement does one case need?

Real judgement: context, relationships, consequences Some — mostly routine with genuine exceptions None — it is mechanical, just tedious

What this looked like in practice

Four of five fixes were not AI fixes. When I hardened an extraction pipeline after a run where most real-world cases failed, the tempting move was to reach for the model: a better prompt, a stronger vision setting, a larger context. In the end, of five root causes, four were fixed deterministically — cleaning the search text before it ever reached the model, widening a candidate set that was truncating valid matches, decoupling an eligibility rule from a signal that was not reliable enough to gate on, and protecting a field from being overwritten by a plausible-looking alternative. Exactly one fix touched the model itself, and it was to make it less creative by pinning it to deterministic output.

That ratio is not unusual, and it is the argument in miniature. Most of what looks like model failure is input handling, candidate selection and business logic wearing a model’s clothes.

A cost-versus-accuracy choice I deliberately did not make. The same pipeline had a setting that would have meaningfully improved text recognition on difficult images, at a materially higher price per call. It was tempting to just turn it on — the output would have looked better immediately.

I did not, and the reasoning matters more than the outcome: this was not an engineering decision. The question “is a percentage point of accuracy worth this much per unit of volume?” is owned by whoever pays the bill and carries the consequence of a wrong answer. My job was to quantify both sides and hand over a decision, not to spend someone else’s money on a default. I documented the trade-off and left it open.

Subscription or API, for a very small business. A small retail operation needed image-based categorisation. The technical options were near-equivalent in quality; the difference was entirely commercial. At their volume, a flat consumer subscription with a human in the loop was cheaper and simpler than metered API calls plus the engineering to run them — and it degraded gracefully when the owner was busy. The correct recommendation was the less impressive-sounding one.

The questions the gate asks

• Not could we use AI here — should we? If the current process already fits, what new risk are we introducing?
• Would a deterministic rule be cheaper, faster and more testable?
• Does this create something a competitor cannot copy — proprietary data, accumulated operational knowledge — or is it a capability anyone can buy next quarter?
• How quickly can it reach production? Time-to-production is itself part of the advantage; a perfect design that lands in eighteen months has lost to a good one that landed in three.
• What does a wrong output cost, and who absorbs it?

How this transfers

The output of this stage is a defensible shortlist with reasons attached — including explicit “no” and “not yet” entries, which are the ones that save the most money. In an enterprise setting I would expect it to be the most contested stage, because a “no” here contradicts someone’s roadmap. That is exactly why it needs to be a documented gate with stated criteria rather than a judgement call made in a corridor.