It is month-end Friday, just after half past six, and a shopper in Durban asks the meal assistant for "a braai for ten tomorrow: boerewors, chicken, pap and chakalaka, two of us are vegetarian, under R1,200". The spinner runs long. Then: "We couldn't build your basket right now. Here is the recipe and a shopping list." That is Part 10's graceful degradation working as designed, but it is happening to more and more shoppers on the busiest evening of the month.
Ayesha Patel, who leads site reliability engineering (SRE), opens the incident channel and asks the SRE assistant: what changed?
Ubuntu Grocer is a composite: a fictional South African grocery retailer whose scenario draws on lessons from real enterprise retail-agent delivery, with illustrative people, systems and figures that describe no single organisation. Part 13 covered the controls at the edge. This part covers the assistant that helps run what sits behind it.
The operating model already exists
CloudNala's AI for Cloud Operations series sets out the general case: start an operations agent read-only, climb an explicit authority ladder from read to write, and separate incident work into evidence, diagnosis, recommendation and remediation, because each tolerates error differently. This part applies it to an incident measured in baskets rather than CPU, where the most dangerous fix is rarely a delete.
The assistant observes an alert on the journey's service level objective (SLO), here the share of conversations ending in an accepted basket, rather than a server metric (see SLOs and error budgets in plain English). It gathers evidence with fixed, tested queries over the journey traces from Part 11. It recommends ranked hypotheses, each with a test, and options, each with a risk and a rollback. A named person approves one exact change, which is executed through the normal pipeline and tracked. The assistant then verifies two things: that baskets recovered, and that hard constraints still hold.
Steps one to three cannot change anything: that is read-only by default.
Friday evening, through the loop
The times below are illustrative.
18:42, evidence. A short summary, with a link behind every fact. Fallback events are concentrated in the ingredient-to-product matcher, whose calls to the catalogue database are mostly waiting for a connection. Model endpoints and the AI gateway look normal. One recent change touches this path: at 17:55 the matcher's autoscaling maximum was raised ahead of the weekend.
18:46, hypotheses. Three. First, connection exhaustion: more matcher replicas, each with its own pool, have pushed total demand past the database's connection limit, the pods-times-pool arithmetic Part 12 says to do at maximum replicas. Test: compare open connections with the limit. Second, a slow vector query after the afternoon catalogue refresh. Test: query time excluding pool wait. Third, gateway quota. Evidence against: no throttling at the gateway.
18:50. Ayesha runs the two tests. Open connections sit at the limit; query time without the wait is normal. The first hypothesis holds.
18:52, options. Roll back the autoscaling change, prepared with its rollback. Raise the database connection limit, recommended but not prepared, because the database team owns its memory and failover effects. Turn on the matcher's degraded mode. And a warning against what the graphs seem to suggest, adding replicas, because each would bring another pool.
18:55. Ayesha approves the rollback, as a named person, through the emergency change route. The assistant triggers the pipeline, which applies it under its own identity, bound to the exact change she saw.
19:05, verify. Pool waits fall and the accepted-basket rate recovers. The assistant samples recent traces and confirms allergen and dietary filter events are still present on every basket, and says what it did not check: whether abandoned baskets will come back. The Durban shopper tries again and gets a basket with boerewors, chicken, and a vegetarian option for two.
The assistant saved time; people kept the decisions.
The fix that looks right and makes it worse
In a cloud estate the frightening mistake is a delete in the wrong account. In a retail journey the dangerous remediations look sensible on a latency graph:
- Scaling out when the constraint is connections. More replicas, more pools, a deeper outage.
- Flushing a cache at peak, so every request falls through to catalogue and pricing at once.
- Skipping a step to recover latency. The step that is easiest to skip is the hard-constraint check, and skipping it breaks the promise to the parent whose child is allergic to peanuts, the kind of hard constraint Part 4 insists is enforced in code.
- Rerouting everything to a smaller model. It relieves capacity and can silently change basket quality.
- Loosening a web application firewall (WAF) rule to make errors stop, which trades a visible false block for an invisible false allow.
None of these is purely an infrastructure decision. Each changes what shoppers are offered or protected from, which is why a retail SRE assistant needs a domain authority map, not only a generic cloud one.
Authority is set per action, before the incident
Ubuntu Grocer's map follows the AI for Cloud Operations ladder, with one retail-specific lane at the bottom: relaxing allergen or dietary rules, changing prices, editing WAF rules and touching checkout are never done through the assistant, even with approval. Those decisions belong to Sipho Nkosi for catalogue rules, Thandeka Mokoena for security and Lindiwe Dube for pricing and the shopper experience.
The assistant never approves its own proposal, and approval is tied to a person's identity, not a channel where anyone could click. During Black Friday and the festive peak, an environment switch drops it to read-only in production, because a wrong change costs most when everyone is busiest.
Autonomous remediation is an authority decision
"Should the assistant fix this automatically next time?" sounds technical. It is a decision about who is accountable for a change no person approved.
The evidence for granting it is a track record: a narrow, reversible class of action approved unchanged almost every time. Ubuntu Grocer has one candidate: turning on the matcher's degraded mode when the fallback rate crosses a threshold. It is designed behaviour, keeps hard constraints and is easily reversed. Even so, the decision to automate it belongs to Lindiwe, because it changes what shoppers see. The gap between agent capability and agent authority is where agent incidents start, and an approve button only counts as oversight if the approver has the evidence and time to refuse, as human-in-the-loop is not a button argues.
Read-only still needs design
Read-only is a starting architecture, not the whole risk. Three things need attention.
The traces contain sensitive data. Shopper messages mention allergies, health conditions and halaal or kosher requirements. Under the Protection of Personal Information Act (POPIA), section 26, health and religious beliefs are special personal information (general information, not legal advice). The assistant reads redacted or minimised traces, not raw conversations.
Logs contain shopper-authored text. Anything a shopper types can end up in a trace, so trace content is untrusted input, never instruction.
Read-only is enforced by identity, not by prompt. The assistant's credentials cannot write, whatever it is told. The OWASP Top 10 for Large Language Model (LLM) Applications names this risk Excessive Agency (LLM06:2025) and recommends minimal permissions and human approval for high-impact actions.
Questions to take into your architecture review
- Does the SRE assistant alert on the journey outcome, or only on infrastructure metrics?
- Are its evidence queries fixed and tested, with a link behind every fact it reports?
- Is its read-only status enforced by the identity it runs under?
- Which actions sit in a "never via the assistant" lane, and who owns each of them?
- Does verification check that hard constraints still hold, and are remediations that would make the journey worse flagged?
- Who decides whether any remediation becomes automatic, and what record supports that decision?
- How are shopper messages redacted before the assistant reads traces, and is trace text treated as untrusted?
- What switches the assistant to read-only during peak trading, and when was it last tested?
Key points
- Read-only by default means the assistant can observe, gather evidence and recommend, but cannot change anything.
- In retail, the dangerous fix is often the sensible-looking one: scaling out, flushing caches, skipping constraint checks or loosening security.
- Authority is set per action before the incident, with a lane of decisions the assistant never makes, even with approval.
- Verify recovery and hard constraints together. A fast basket that ignores an allergy is not a recovery.
- Automating a remediation is an authority decision owned by the business, supported by a record of approvals.
CloudNala's point of view
An SRE assistant earns its keep in the first minutes of evidence, not the last click of the fix. In reviews of retail-agent programmes we see teams granting write access because a demo restart worked, while evidence queries are still ad hoc and verification ignores hard constraints. Make the read side excellent first. Write access is earned one action at a time, and some actions are never earned.
Next: Part 15, The production-ready agentic retail checklist. All parts are on the series page.
Background reading: OWASP Top 10 for LLM Applications, LLM06:2025 Excessive Agency and POPIA section 26. Both checked on 5 November 2026.
How CloudNala can help
CloudNala helps retail platform and SRE teams design AI-assisted operations around the shopping journey: journey-level SLO alerts, fixed evidence queries over your traces, a domain authority map that names what the assistant may observe, recommend, prepare or never touch, verification that checks hard constraints as well as latency, and an approval path your change board can sign off before the next peak.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agentic Retail Review or write to us at consult@cloudnala.co.za