When the board of Rooibos Agri Finance, a fictional South African provider of seasonal input finance to farmers, asked for a quarterly view of how its AI was performing, the first draft of the report had one number on it: "AI accuracy: 91%". Its figures are illustrative. The number came from the engineering team, and nobody could say what it measured. It turned out to be the share of applications in a test set where the service's final outcome matched what a senior credit officer would have decided. It was a reasonable thing to measure. It was also useless for running the service, because it could not say which of the five capabilities in the service was responsible for the 9% that went wrong.
Part 1 of this series mapped the capabilities in Rooibos's service: explicit rules, a repayment-risk model, image and document analysis, a generative explainer and a bounded document-collecting agent. Part 2 showed how they combine and where the seams fail. This final part deals with evidence. Each capability fails in its own way, so each needs its own evaluation, with its own owner and its own trigger for running it again.
Why one number fails
An end-to-end score is useful as a final check. As the only measure, it has three problems.
- It cannot locate a fault. A wrong outcome may come from a rule, a misread document, a poorly calibrated risk estimate or an agent that never collected the right paper. The score says something is wrong, not where.
- It averages away the failures that matter. A service can be 91% right overall and much worse for applicants with handwritten documents or for a region the risk model saw little of.
- It hides layers that do not affect the outcome. The generative explanation does not change the decision, so an outcome score cannot see a misleading explanation at all. The Clanwilliam complaint in Part 2 would have scored as a success.
Test the rule, test the model, test the whole decision sets out a layered testing approach for public-sector decision systems: rule tests, model tests, integration tests, scenarios, fairness and security, and outcome monitoring. This part takes a different cut. It asks what evaluation each kind of AI capability needs, because the right questions for a statistical model, a generative model and an agent are fundamentally different.
What to measure, layer by layer
Explicit rules: is the rule what policy says?
Rules are ordinary software, and the referenced article covers their testing well: worked examples from the policy owner, boundary cases and conflict checks. The question being answered is fidelity to policy, and the evidence is a passing test suite tied to a rule version. No statistical measure applies; a rule is right or wrong for a given case.
Statistical prediction: is the estimate useful and honest?
The repayment-risk model needs evidence that rules and generative models do not.
- Discriminatory power (ranking). Does the model rank applicants usefully, so that those it rates as riskier really do repay less often? Credit teams usually summarise this with a Gini coefficient or the area under the ROC curve (AUC).
- Calibration. When it says a group has a 20% chance of not repaying, do roughly 20% of them not repay? A model can rank well and still be badly calibrated, and Rooibos's thresholds depend on calibration.
- Out-of-time validation. Does it hold up on applications from a later period than it was trained on? Seasonal finance is exposed to drought years, price shocks and policy changes that a model trained on good years has never seen.
- Stability. Has the population of applicants shifted away from the population the model learned from? This is measured continuously, not only at release.
- Performance by segment. Discriminatory power and calibration for each segment the organisation is permitted to analyse, such as region, farm size or product, so that a model that works on average does not quietly fail one group.
The owner is the credit risk modelling team, and model risk holds independent challenge.
Perception: are the facts read correctly, and does it know when it cannot read them?
- Accuracy per field. How often is the title-deed number right? The registered owner? The planted area? Overall accuracy is meaningless when one field drives the decision.
- Accuracy by input condition. Clear scans versus phone photographs, typed versus handwritten, good light versus poor. Rooibos found its photograph check was reliable in daylight and poor at dusk, when many farmers take the photographs.
- Abstention. When the input is unreadable, does the component say so, or does it guess? A good perception component rejects what it cannot read. Measure the rate of confident wrong answers separately from the rate of honest rejections.
Generation: is the explanation faithful, complete and readable?
Single-label accuracy does not fit generated text. The questions are different.
- Faithfulness. Does the explanation state only reasons that are in the case record? This is the test that would have caught the Clanwilliam failure.
- Completeness. Does it include every reason the record contains?
- Readability and language. Can the intended reader understand it, in the language they used to apply?
- Harm. Does it avoid inappropriate, discriminatory or misleading content?
These are measured with a rubric scored by people on a sample, and partly automated where the record allows (for example, checking that every reason code appears and no other). NIST's Generative AI Profile (NIST AI 600-1, released on 26 July 2024) is a useful reference for the risks specific to generative systems.
Bounded agent: did it do the task, within scope, and stop properly?
An agent's output is a sequence of actions, so its evaluation is about behaviour across a run.
- Task completion. Did the application file end up complete?
- Scope adherence. Did it use only its permitted tools, ask only for documents on the required list and never touch the credit record?
- Stopping. Did it stop when the file was complete, when the applicant stopped responding or when it reached its attempt limit?
- Burden on the applicant. How many messages did it send to get one document? An agent that completes files by pestering farmers is failing.
- Trace review. A sample of full runs reviewed by an operations lead, because some failures only show up in the sequence.
You do not control the agent until you control the loop covers the controls that make stopping and scope testable.
The whole service: are the outcomes right and fair?
The end-to-end check stays, now as one measure among several: agreement with senior officers on a reference set, officer override rates, appeal rates and appeal success rates. It answers whether the combination works. The layer evaluations answer why when it does not.
Metrics that do not transfer
Teams often reuse the metric they know best on a layer it does not fit.
| Metric | Fits | Misleads when applied to |
|---|---|---|
| Single-label accuracy | Perception fields; rule outcomes on test cases | Generated text, where many wordings are correct, so single-label accuracy does not fit |
| Fluency or readability | Generated text | Rules and risk estimates, where fluency means nothing |
| Calibration | Risk estimates | Rules |
| Task completion | Agents | Used alone, without scope adherence and burden |
| End-to-end agreement | The whole service | Any single layer, because faults cannot be located |
Which change triggers which evaluation
Evaluation is not a release-time event. Each layer must be re-evaluated when something it depends on changes. Rooibos keeps a trigger table.
| Change | Re-evaluate |
|---|---|
| Credit policy or rule change | Rule tests; end-to-end scenarios; explanation faithfulness for affected reason codes |
| Risk model retrained | Discriminatory power, calibration, segment performance; thresholds re-approved; end-to-end |
| Population shift detected | Risk model calibration and segment performance; consider thresholds |
| Document or photograph model updated | Per-field accuracy, by condition; abstention; end-to-end |
| Generative model or prompt changed | Faithfulness, completeness, readability, harm |
| Agent tools, limits or prompt changed | Scope adherence, stopping, burden, trace sample |
| New region or product added | Every layer, with segment results for the new population |
The table is the practical link between evaluation and the change process, and it is what turns occasional testing into the habit described in Continuous evals are tests for AI systems. A provider's model update or a retrain is a material change, and the table says which evidence has to be refreshed before it ships.
The board view, rebuilt
Rooibos's revised quarterly report has one line per capability: what it does, the owner, the headline evidence (calibration for the risk model, per-field accuracy for documents, faithfulness for explanations, scope adherence and burden for the agent, test coverage for rules), the date the evidence was last refreshed, and the trigger events since then. The end-to-end agreement figure, override rate and appeal rate sit underneath as the service view.
The new report is longer than one number and much shorter than the old engineering dashboards. Its main effect has been on questions: board members now ask "when was the risk model last validated on this season's applicants?" instead of "is the AI accurate?". For a wider view of what boards need, see What an AI value dashboard should show the board.
Key points
- One accuracy number cannot locate a fault, hides segment failures and cannot see layers that do not change the outcome.
- Rules are tested for fidelity to policy; risk models for discrimination, calibration, out-of-time validity, stability and segment performance; perception for per-field accuracy by condition and honest abstention.
- Generated text is evaluated for faithfulness, completeness, readability and harm, not single-label accuracy; agents for task completion, scope adherence, stopping and burden.
- Do not reuse a metric on a layer it does not fit.
- Keep a trigger table so that each change refreshes the evidence for the layers it affects, and report each layer separately to the board.
This is the final part of AI Is a Stack, Not a Single Technology.
How CloudNala can help
CloudNala runs AI decision reviews that include an evaluation design for every layer of a service. We identify which capabilities your service uses, define the evidence each one needs and who owns it, build the re-evaluation trigger table into your change process, and design a board view that reports each layer honestly, so that a wrong outcome can be traced to its cause.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Decision Review or write to us at consult@cloudnala.co.za