When six of seven runs failed

The pipeline took a photograph of a physical item and produced structured metadata: identify the title from the cover, find the matching record in a public catalogue, and return the canonical entry. Text recognition, candidate retrieval, ranking, eligibility rules, and a structured response. It had a full automated test suite, and the suite was green.

Then it met seven real photographs, and six of them came back wrong.

This is the case I would show first to anyone who wants to know how I work, because everything interesting about AI systems lives in that gap between a green suite and reality.

Why the suite was green and the system was broken

The tests were checking that the code did what the code was written to do. They were not checking whether the system produced the right answer for a photograph taken on a kitchen table under a ceiling light. Every failure lived in the space between those two statements: input quality, candidate truncation, an eligibility rule that was defensible in isolation, and a model that was free to be creative about a field where creativity is a defect.

The five root causes

I refused to treat “six failed” as one problem. Each case was diagnosed to a cause before anything was changed.

1. Unclean search text. What came off the cover carried artefacts — punctuation, badge text, formatting noise — straight into the catalogue query. The query was therefore searching for a string that did not exist anywhere. Fixed deterministically, before the query.
2. A candidate set truncated too early. For editions whose covers carry an extra badge, the correct record consistently fell outside the retained candidate window. The ranking was fine; it was never shown the right row. Fixed by widening the candidate set for that shape of input.
3. An eligibility rule gated on an unreliable signal. Image-derived acceptance had been ANDed with a second, weaker signal. Whenever the weaker one wobbled, a correct match was discarded. Fixed by decoupling: image-based eligibility now depends only on the signal it can actually stand on.
4. A non-deterministic model on a deterministic task. Extraction was running with sampling enabled, so the same photograph could yield different readings. Pinned to deterministic output.
5. A protected field being overwritten. The model would occasionally replace a correctly extracted creator name with a more famous, plausible one — the classic failure where fluency beats fidelity. Fixed by protecting the field explicitly rather than hoping the prompt would hold.

Four of the five fixes are not AI fixes. That ratio is the most useful thing on this page.

Six failures, five causes — and one that stayed failed

Each failure was diagnosed to a cause before anything was changed. Note how often the first guess — "the model is not good enough" — was wrong.

1. 1 The catalogue lookup returned nothing at all for several items. Input handling

   First guess: The model misread the cover.

   Actual root cause: The extracted text carried artefacts — punctuation, badge fragments, formatting noise — straight into the query. The system was searching for a string that exists nowhere.

   Fix: Clean and normalise the search text deterministically, before the query is built.

2. 2 One family of editions consistently failed, others were fine. Retrieval

   First guess: Ranking is weighting the wrong signal.

   Actual root cause: Those editions carry an extra badge on the cover, which pushed the correct record outside the retained candidate window. The ranking never saw the right row.

   Fix: Widen the candidate set for that shape of input, so ranking is given the chance to be right.

3. 3 Correct matches were being discarded as ineligible. Business logic

   First guess: The eligibility threshold is too strict.

   Actual root cause: Image-derived acceptance had been combined with a second, weaker signal using AND. Whenever the weaker signal wobbled — which it did often — a correct match was thrown away.

   Fix: Decouple them: image-based eligibility now depends only on the signal it can stand on.

4. 4 The same photograph produced different readings on different runs. Model configuration

   First guess: Image quality is borderline.

   Actual root cause: Extraction was running with sampling enabled — a creative setting on a task that has exactly one correct answer.

   Fix: Pin the extraction to deterministic output.

5. 5 A correctly read creator name was replaced by a different, more famous one. Output integrity

   First guess: Bad source data.

   Actual root cause: Classic fluency-over-fidelity: given a slightly unusual name, the model substituted a well-known one that fit the context better. The output looked more plausible and was wrong.

   Fix: Protect the field explicitly rather than relying on the prompt to hold under pressure.

The seventh case: left failing, on purpose

One item never resolved. The record was not in the source catalogue — there was no correct answer to retrieve. Tuning the system until it produced something would have taught it to invent a confident answer for exactly the inputs where being wrong is most expensive. It is documented as a source-data gap, not a defect.

Verification: every fix was checked by invoking the deployed pipeline and reading the actual execution output, then re-run against a thirty-item reference set at top-1 and top-5. 30 / 30, zero regressions across all five changes.

The seventh case, which stayed failed

One item never resolved. I spent a while trying to make it work before checking the assumption underneath: the record simply was not in the source catalogue. There was no correct answer to retrieve.

So I left it failing, and documented it as a source-data gap rather than a defect.

This is the decision I care most about in the whole case. Every instinct — and every incentive, if someone is watching a pass rate — pushes toward making the number go green. But a system tuned to produce a confident answer where no answer exists is not a better system; it is a system that has learned to hallucinate on exactly the inputs where being wrong is most expensive. The correct behaviour was to fail, and my job was to say so.

How the fixes were proved

Fixing is easy. Proving you fixed it without breaking something else is the job.

• A thirty-item reference set built from real cases, with the expected canonical record for each, measured at top-1 and top-5. The set predates the failures, which is what makes it evidence rather than a post-hoc justification.
• Behavioural verification, not API inspection. I had learned the hard way on this stack that inspecting configuration can show one version while the running system executes another. So each fix was verified by invoking the deployed pipeline and reading the actual execution output — checking a computed value inside the chain to confirm which version had run.
• Full regression after every change, not only at the end: thirty of thirty on the reference set, top-1 and top-5, with zero regressions across the five fixes.
• Diagnosis inside the chain, not at the edges. The eligibility bug was invisible from the final response. Finding it meant reading the intermediate outputs node by node. The final answer tells you that it failed; only the intermediate state tells you why.

What I would put in front of a stakeholder

A pass rate is not an acceptance criterion. What a release decision needs is: which categories of input are covered by the reference set, what the measured rate is per category, which failure modes are known and accepted, which are blocking, and what will catch a silent regression when the model version changes next month.

“Eighty-six percent correct” is not a result. It is the beginning of a conversation about which fourteen percent, how you would notice, and who carries it when it happens.

How this transfers

Directly, and this is the part of enterprise AI I find most consistently missing. Leadership frameworks handle whether a use case is worth doing and whether it paid back. Between those sits the release decision — is the output good enough to put in front of a customer — and it needs exactly this machinery: an agreed acceptance bar written before the build, a reference set someone owns, regression on every change, and root-cause discipline that is allowed to conclude “the data is wrong” rather than tuning until the number looks better.

I spent twelve years making that call for software that had to ship. The system is probabilistic now; the question has not changed.