The launch review for the meal assistant is on a Thursday afternoon, a week before the planned staff pilot. The status slide is mostly green. Lindiwe Dube, head of digital commerce, looks at it for a moment and asks for something else: "Show me one basket." She picks the request the team has used since the first demo: supper for four tonight, one child is allergic to peanuts, keep it under R400, we already have rice.
Ruan Botha, the AI engineering lead, runs it with the primary model forced unavailable, the condition the slide marks as tested. The fallback route returns a sensible meal plan and a tidy basket. The trace shows the peanut-allergy filter never ran. The workflow was right; the routing was not. The fallback deployment in the second location ran an older routing configuration that sent plans straight to the matcher, around the workflow's constraint step. The slide was not lying about the tests. The tests had checked that fallback produced a basket, not that it produced a safe one.
Ubuntu Grocer is a composite: a fictional South African grocery retailer whose scenario draws on lessons from real enterprise retail-agent delivery, with illustrative people, systems and figures that describe no single organisation. Part 14 covered the operations assistant. This final part turns the whole series into one review you can run on your own programme.
Three rules for using the checklist
The checklist has twelve dimensions. Each has a description of what good looks like, the evidence to ask for and a red flag. Three rules make it work.
Evidence, not assertion. Every status points to something a reviewer can open: a signed decision-rights map, a load-test report, a trace, an evaluation run. "The team is confident" is not evidence. This is the same move CloudNala's definition of production-ready makes in general, and why a passed test is not proof of production readiness.
Red flags first, and never averaged. A single red flag decides a dimension's status, however good the rest of its evidence looks, and some red flags hold the launch on their own.
Show it on one basket. Almost every dimension can be demonstrated on a single traced journey: the outcome it produced, the constraints applied, the models it used, the tokens it spent, the fallback it took, the edge decisions it passed. If a claim cannot be shown on one real basket, record it as partial.
The checklist
Part numbers link to the article that explains the mechanism behind each row. SLO is a service level objective, SKU a stock-keeping unit (one sellable product), and WAF the web application firewall at the edge.
| Dimension | What good looks like | Evidence to ask for | Red flag | Parts |
|---|---|---|---|---|
| Outcome | Success is measured on baskets: valid basket rate, line acceptance and plan-to-checkout conversion, with a journey SLO and a named business owner | The measures in writing; the SLO; a baseline from today's manual journey | Success is counted in conversations or answers | 1, 2 |
| Data | Product identity, allergens, dietary marks, pack sizes and local-language synonyms are maintained by named owners; recipe lines map to SKUs through an explicit intermediate step; a product with missing allergen data is excluded from allergy baskets (fail closed) | Attribute coverage by category; owner list; mapping tests for units and pack sizes | Allergen attributes inferred by the model, or a missing value treated as safe | 3, 5 |
| Authority | A decision-rights map says who may search, recommend, substitute, change the basket and check out | The signed map; tests showing the basket agent cannot reach checkout | The agent can call checkout or payment | 6 |
| Workflow | A deterministic journey calls bounded agent steps; hard constraints filter in code at a fixed step | Workflow definition; a test proving a hard-constraint violation cannot reach the basket | Allergies or halaal rules exist only in the prompt | 4, 6 |
| Capacity | The knee of the latency curve is known for realistic journeys, and quotas fit peak demand with headroom | Journey load-test report with scenario mix; a token capacity model; agreed quotas | Load tested with one prompt, or with the edge bypassed | 7, 8, 9 |
| Cost | Cost per accepted basket is known by step and model, and has a budget and an owner | Token and cost telemetry joined to journey outcomes | Only a monthly model bill exists | 9, 10 |
| Observability | One journey ID links front end, gateway, models, catalogue, cache, database and basket | One real basket's trace, opened in the meeting | Teams diagnose by comparing separate dashboards | 11 |
| Resilience | Fallback degrades one rung at a time, the constraint filter runs on every rung and route, and constraint dependencies fail closed; pods × pool size at maximum replicas and surge fits the database limit; configuration matches across locations | Forced-failover test that asserts the filter event in the trace; the connection budget; a drift check | A fallback rung or route that skips the constraint filter, or readiness that reports healthy before dependencies are | 10, 12 |
| Security | Edge rules have should-allow and should-deny tests, roll out in stages and roll back quickly | The two-way test suite; rollout record; blocked-legitimate metric | Rules changed directly in production; deny-list access rules on internal pages | 13 |
| Evaluation | Evaluation sets cover allergens, budgets, pack sizes, multilingual requests and long conversations, and gate releases | Evaluation results per release; scenario generator config | Evaluated on demo prompts only | 2, 4, 7 |
| Human approval | The shopper reviews and confirms every basket; operational changes need a named approver | Basket review screen; approval records bound to exact changes | Basket changes or checkout without explicit shopper action | 2, 6, 14 |
| Operational ownership | Every component, including each agent step and the catalogue data, has an on-call owner and a runbook | Ownership register; runbooks; the SRE assistant's authority map | A component that "belongs to the AI team" with no on-call rota | 5, 12, 14 |
Four red flags hold a launch on their own, whatever else is green: a hard constraint that can be bypassed on any path, agent access to checkout or payment, a fallback that drops constraints, and a component nobody operates. Each one means a shopper can be harmed or a failure can go unanswered, and no score elsewhere compensates.
The data row deserves one more note. Allergy, health and religious dietary information is special personal information under section 26 of the Protection of Personal Information Act (POPIA), so the data evidence should include how it is minimised, where it is stored and who can read it. That is general information, not legal advice; take it to your information officer.
The fourteen mechanisms behind the checklist
The twelve rows draw on fourteen mechanisms, one per part.
The value journey. Part 1: the model is one box in a stack, and value appears only when the whole stack works. Part 2: the outcome is a valid basket the shopper accepts, not the answer. Part 3: recipes and catalogues need an explicit mapping from ingredient line to chosen SKU and quantity.
Catalogue and decisions. Part 4: hard constraints filter in code; soft preferences rank. Part 5: catalogue quality controls answer quality. Part 6: decide the boundary and the decision rights before building the agent.
Production engineering. Part 7: load-test varied journeys, not one endpoint. Part 8: the bottleneck moves as concurrency rises; find the knee. Part 9: tokens per journey are a capacity input. Part 10: route by task and degrade without dropping hard constraints. Part 11: one journey ID across every dependency. Part 12: probes, pools and configuration decide production.
Governance and operations. Part 13: security rules fail both ways and roll out in stages. Part 14: operations assistants start read-only and earn authority per action.
Readiness is staged too
Not every row must be complete for a staff pilot. Some must be complete before anyone uses the assistant, and the rest accumulate as exposure grows.
The first gate, a staff pilot, is short but strict: the outcome is defined, hard constraints are in code, the authority map is signed, no agent can reach checkout, every journey is traced, the fallback is proven to keep the rules, and every component has an owner. Those last two are launch-holding red flags, so they cannot wait. Later gates, from invited shoppers onwards, add what more exposure demands: evaluation in the languages shoppers really use, a load test through the edge and two-way security tests. The last gate is a retest at the shape of a real peak, such as month-end payday or the festive season, because the knee found in a quiet week is not the knee on a Sunday afternoon in December.
Ubuntu Grocer's review
The illustrative results above are where Ubuntu Grocer's Thursday review ended. Six dimensions were ready. Four were partial, each with an owner and a date: allergen attributes still missing for some categories, so those products were excluded from allergy baskets until fixed, a load test not repeated since the matcher changed, no should-allow security tests and no multilingual evaluation set. Two red flags held the launch: the fallback route that skipped the allergy filter, and the ingredient-to-product matcher, which had been built by the AI team and handed to nobody for production support.
Neither fix was large. The drifted routing configuration was rebuilt from the same source as the primary, a drift check joined the pipeline, and the forced-failover test now asserts that the filter event appears in the trace, not merely that a basket came back. The matcher got an owner, a runbook and a place in the on-call rota. The peanut basket was rerun, the filter appeared in the trace, and the staff pilot went ahead.
Each owner signed their own rows: Sipho Nkosi for data, Pieter van Wyk for workflow and resilience, Thandeka Mokoena for security, Ayesha Patel for operations, Ruan Botha for evaluation, and Lindiwe for the outcome and the launch decision. Nobody signed the whole sheet alone. The review is repeated before each peak and after any model, prompt or routing change, because if you cannot define good you cannot ship the model change.
Questions to take into your architecture review
- Can every status on your readiness slide be traced to evidence a reviewer can open today?
- Which red flags would hold your launch on their own, and is that list written down?
- Can you show one real basket's trace, end to end, including the edge and every model call?
- Does any route through the system, including fallback, skip a hard-constraint check?
- Who signs each dimension, and does anyone sign the whole review alone?
- What must be true before the first shopper, and what can wait for a later gate?
- When was the review last repeated, and what change since then should trigger it again?
- Which component would nobody answer for at two in the morning on Black Friday?
Key points
- Production readiness for a retail agent is twelve dimensions of evidence, not one adjective.
- Check red flags first and never average them away. Some hold the launch on their own.
- Demonstrate each dimension on one real, traced basket; a claim that cannot be shown is partial.
- Stage readiness with exposure, but treat hard constraints, authority and tracing as day-one requirements.
- Owners sign their own rows, and the review is repeated before peaks and after model or routing changes.
CloudNala's point of view
The most useful question in any launch review is "show me one basket". It cuts through slides, because a single trace reveals whether the outcome, constraints, routing, cost and ownership hold together. In reviews of retail-agent programmes we rarely find a weak model; we find a fallback nobody tested for safety, a component nobody operates, or a security rule nobody tested against real shoppers. The agent really is the smallest part of the system, and this checklist is how you check the rest.
This is the final part. All parts are on the series page. To start again, read Part 1, The retail agent is the smallest part of the system.
How CloudNala can help
CloudNala runs this checklist as an Agentic Retail Review: we collect the evidence for each of the twelve dimensions, trace real baskets through your journey, identify red flags that should hold a launch, and leave you with a scorecard whose rows have named owners, dates and the evidence behind them. It works before a first launch, before a peak and after a major model or routing change.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agentic Retail Review or write to us at consult@cloudnala.co.za