Generating at scale, and validating every piece

The requirement was a large, bilingual bank of assessment items derived from existing structured test material. The volume made manual authoring impractical; the use — certification — made quality non-negotiable. An item that is ambiguous, factually wrong, or trivially guessable does not just fail to help, it actively damages trust in the whole programme.

That combination is the interesting one. Generative AI is very good at volume and indifferent to quality unless quality is engineered into the system around it.

Quality as rules, not hope

The first thing I did was stop treating quality as a property of the prompt and start treating it as a specification the output must satisfy. Three rules did most of the work:

• Scenario constraint. Every item had to describe a concrete, realistic working situation rather than ask an abstract definitional question. This one rule eliminated the largest class of useless output — items that test whether you have memorised a phrase.
• Answer-length ratio. The correct answer could not be substantially longer than the average distractor. Without this, models reliably produce a detailed, careful correct answer surrounded by short throwaways, and the item becomes solvable by shape alone without reading it.
• Distractor plausibility. Wrong answers had to be wrong for a reason a competent person could believe. Obviously-wrong distractors make an item look rigorous while measuring nothing.

Writing these down as enforceable constraints, rather than as guidance in a prompt, is what made the output usable at volume.

Generation and validation as separate systems

Generation ran as a fan-out: the source material was split into batches and processed in parallel, which turned a sequential authoring problem into a throughput one.

Validation was deliberately built as its own pass rather than folded into generation, because a model that has just produced something is a poor judge of it. The check ran from two perspectives in one structured pass — one asking whether a candidate taking the assessment could answer this correctly and fairly, the other asking whether an examiner would accept it as valid — and covered factual correctness, language quality in both languages, domain specificity, the answer-length rule, and structural validity of the output format.

Repair, split by failure type

The part I would emphasise to anyone building similar systems is that not all errors deserve the same repair mechanism, and using an agent for everything is expensive and unpredictable.

• Systematic, mechanical errors — malformed structure, escaping, encoding, formatting drift, repeated typos — were fixed programmatically. These are deterministic problems and code fixes them identically every time, which also means the fix is testable.
• Semantic errors — a platform-specific detail attributed to the wrong platform, an answer whose logic had been inverted, a distractor that was accidentally also correct — went to an agent pass, because judging them requires understanding rather than pattern matching.

The split matters commercially as well as technically: routing mechanical errors to a model is paying language-model prices for a task a regular expression does better.

The outcome that matters

Several hundred items across two languages reached the framework with zero lines of manual editing — not because the first generation was perfect, but because every failure had a defined route to repair, and the criteria for “acceptable” existed before generation started.

There is a second, less obvious result. The same project involved building the surrounding software itself with AI as a development partner. So the finished thing demonstrates two distinct capabilities in one place: AI used to build the tool, and AI used to produce the content the tool runs on. I mention that because it is the honest description of the work, and because the two skills are frequently confused.

How this transfers

Any enterprise generating content at volume — knowledge base articles, structured test material, policy summaries, customer communications — faces exactly this problem, and usually solves it by putting a person in front of every item. That is not automation; it is relocation, and it caps the value at whatever review capacity exists.

The alternative is the shape above: quality expressed as enforceable rules, an independent validation pass, repair routed by failure type, and human attention spent on the items that actually need judgement. The generated output is a draft. The system around it is the product.