There is a tempting shortcut in AI-assisted decision-making. Rather than going through the work of making rules explicit, give a capable model the full policy, the case documents and a good prompt, and ask it what the outcome should be. Modern models are often right. They explain themselves fluently. In a demonstration, the shortcut looks like the whole solution.
It is not, and the reason is not that models are bad at reading. It is that a model asked to decide is doing three different jobs at once: working out the facts of the case, working out what the rules mean and applying one to the other. When it gets an answer wrong, nobody can tell which job failed. When it gets an answer right, nobody can show that it will do the same on the next similar case. And if the model's reading of a rule becomes the effective rule, then the rule is whatever the current model version happens to infer, which changes when the model changes.
This part sets out the split Amatola Bursary Agency, the fictional agency in this series, uses instead.
Three jobs, three owners
The model reads and writes. It extracts facts from documents, flags evidence that looks incomplete or inconsistent and drafts reasons in plain language. These are tasks where probabilistic judgement is appropriate, where errors can be caught by a person and where the model adds real value: reading forty scanned pages consistently is tedious work for people.
The rules engine applies approved rules. It takes confirmed facts and the rule version in force, and returns an outcome with a trace of which conditions applied. It is deterministic: the same inputs and rule version always give the same result. It can refuse, returning "cannot decide" when inputs are missing or rules conflict, as covered in Part 3.
The official holds authority. The official confirms the facts the model extracted, exercises any discretion the rules give, decides and signs. In a public body, the official acts under a legal delegation. In a bank or insurer, under an approved mandate. Either way, the authority is a person's, and so is the accountability.
The handovers between these jobs are where the design work lies.
Handover one: extracted facts must be confirmed
The model's extraction is a proposal, not a fact. At Amatola, every extracted field that feeds a rule carries its source location (which document, which page) and a confidence signal from the extraction step. The official's screen shows the extracted value next to the source image. Fields below a threshold, or flagged as inconsistent with other documents, must be confirmed individually. Others can be confirmed in bulk, but the confirmation is recorded either way.
Only confirmed facts reach the rules engine. If a fact is unconfirmed, the engine receives it as missing and returns "cannot decide" for any rule that depends on it. That is deliberately inconvenient. It stops an extraction error from flowing silently into an outcome.
The thresholds belong to the action, not to the model: the bar for confirming an income figure that determines eligibility is higher than for a field used only for reporting. The Jev thresholds, evals and human review article develops this point for typed decision services.
Handover two: the engine's trace becomes the explanation
When the engine returns an outcome, it returns the trace too: which rule, which version, which conditions were met or not met, which exception fired. The model drafts reasons from that trace and the confirmed facts, not from the raw policy documents. It is explaining a decision already made, not making one.
This changes what can go wrong. A model drafting from a trace can still write a clumsy sentence or miss a nuance. It is much less likely to cite a rule that did not apply, because the rules it may cite are given to it. The official reviews the draft, and the final reasons are recorded in the case file from Part 4.
Handover three: discretion stays with the official
Many rules leave room for judgement. Amatola's policy allows the programme manager to waive the academic results requirement in "exceptional circumstances", such as a serious illness during the final school year. That clause cannot be turned into a decision table. It is discretion by design.
The rules engine handles this by returning a distinct outcome: "eligible subject to discretion" or "ineligible unless discretion is exercised", naming the discretionary clause. The model can summarise the circumstances the applicant described and point to relevant evidence. It does not recommend whether to exercise the discretion. The official decides, records reasons and the decision is reviewable like any other.
There is a reason to be strict here. Discretion is where fairness and consistency are hardest to maintain, and where a model's suggestion carries most weight with a busy official. If the model recommends, the recommendation tends to become the decision. Keeping the model to summarising evidence protects the official's judgement rather than replacing it.
What about using a model to interpret the rules?
A reasonable objection: models are good at reading legal text. Why not use one to interpret ambiguous clauses?
Use it to help, not to decide. A model is a useful drafting partner when the policy owner is resolving an interpretation question: it can set out readings of the clause, find related provisions and point to inconsistencies. That happens once, at design time, and the result is an approved interpretation record that the engine applies consistently. What the model must not do is interpret the clause afresh for each case at run time, because then the effective rule varies case by case and changes with every model update.
The same principle applies to retrieval. A model can retrieve policy text to help a reviewer understand a case. The retrieved text informs a person; it does not override the registered rule. If the retrieved text and the engine's rule disagree, that is a register defect to fix, not a choice for the model.
Signs the model has become the law
- Outcomes are produced by a prompt that includes the policy text, with no separate rules engine.
- The same case submitted twice can produce a different outcome.
- A model upgrade changed approval rates and nobody can say which rule changed.
- Reasons letters cite policy clauses that the reviewer cannot find in the rule trace.
- Officials approve recommendations without seeing the extracted facts against their sources.
- Discretionary decisions arrive with a model recommendation attached.
Each of these is fixable without abandoning AI. The fix is to give the model the jobs it does well and take away the job it should not have.
The South African public-sector point
For public bodies, there is a further reason to keep this split. Administrative action has to be lawful, reasonable and procedurally fair, and authority to decide is conferred on officials by law or delegation. A system design in which a model effectively decides and an official signs makes it harder to show that the person with legal authority actually exercised it. Human-in-the-loop is not administrative justice examines this in the government context. This is general information, not legal advice.
Next: Part 7, Test the rule, test the model, test the whole decision. All parts are on the series page.
How CloudNala can help
CloudNala helps teams take an AI-assisted decision process that relies on a model to decide and split it into extraction, rules and authority. We design the confirmation screen and thresholds, the rules engine and its trace, the reasons pipeline that drafts from the trace, and the handling of discretionary clauses, so that each part can be tested and each decision can be defended.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Decision Review or write to us at consult@cloudnala.co.za