At twenty to five on the last Sunday of the month, a shopper in Polokwane types into the meal assistant: "Supper for six tonight, halaal please, my mother doesn't eat beef. Maybe a chicken curry with samp, and add amasi and some morogo if you have it. Under R550, we've got 2 kg of maize meal already." She presses send and gets "Something went wrong. Please try again." She tries a shorter version, gets the same error, and closes the app.
Nothing in the agent had failed. The request never got past the edge. A rule in the web application firewall (WAF), tightened the week before, had decided that a long message full of apostrophes and mixed languages looked like an attack.
Ubuntu Grocer is a composite: a fictional South African grocery retailer whose scenario draws on lessons from real enterprise retail-agent delivery, with illustrative people, systems and figures that describe no single organisation. Part 12 covered readiness probes and connection pools. This part opens the governance-and-operations arc with the controls in front of all of it.
A security rule is a decision about customers
Thandeka Mokoena, Ubuntu Grocer's security architect, describes every WAF rule as a small classifier. Like any classifier, it can be wrong in two directions.
A false block refuses a request that should have been served. It is expensive and badly disguised: the shopper sees a generic error, the agent team sees fewer completed baskets, and the first suspect is the model. A false allow lets through a request that should have been stopped. Nothing breaks, so nobody complains about a door left open.
Security dashboards count blocks; product dashboards never see the edge, so the useful review question is not "is the WAF on?" but "who watches each kind of mistake?"
Failure one: the inverted rule that failed open
Ubuntu Grocer's backend-for-frontend (BFF) serves an internal operations page. The protecting rule was meant to read "block unless the source is the operations network". With the negation on the wrong side it became "block these listed sources": an allow-list turned into a deny-list.
Allow-lists fail closed: anything nobody thought of is refused. Deny-lists fail open: anything nobody thought of gets in, and that grows every time the network changes. The negation is the easy part to invert. This one passed the only test anyone ran: an engineer opening the page from the office.
A review found the gap, not an incident, by running the test nobody had written: the same request from outside. The lessons: read every rule aloud as a sentence; test it from outside, where an unwanted caller would stand; and never let the WAF be the only control on an internal page. The BFF now authenticates staff itself, in line with a practical zero-trust approach, and the flow-matrix discipline from network rules applies at the edge too.
Failure two: the rule that could not read a shopping list
The Polokwane shopper's message is ordinary, and that is the problem. Conversational traffic has exactly the features generic attack signatures look for:
- Punctuated free text. Apostrophes in "doesn't" and "Ma's bobotie", and the typographic quotes phone keyboards insert.
- Text beyond plain ASCII. Crème fraîche, emoji, isiZulu, Sesotho or Afrikaans words.
- Encoded bodies. The text arrives inside JSON. Escape sequences and quotes either survive into inspection, or decoding produces strings the hand-written tests never covered. Depending on configuration, bodies over the inspection size limit are either blocked or passed uninspected.
- Growing length. If recent turns travel with each request, a rule that never fires on the first message can fire on the fifth. Single-prompt tests miss this, which is why Part 7's journey load test must pass through the edge.
The trigger was a well-meant tightening. The widely used OWASP Core Rule Set (CRS) has four paranoia levels, and its documentation says higher levels detect more attacks at the cost of more false positives. Raising the level site-wide after an advisory was a reasonable instinct, but it was a change to the customer journey, made without the journey's tests.
It was also invisible. A blocked request never reaches the BFF, so the journey trace from Part 11 had a hole where the failure was, and for a first message the BFF-created journey ID did not yet exist. The app now sends a client request ID that the edge logs and shows a short reference on the error screen, so support can join a complaint to an edge decision.
Test cases in both directions
Both failures have the same fix: treat rule changes as code, with a test suite covering both directions.
| Test | Direction | Example | Expected |
|---|---|---|---|
| Long multilingual list, fifth turn | Should allow | The Polokwane message with four earlier turns | Served |
| Accented and punctuated product names | Should allow | Crème fraîche, apostrophes in a recipe name | Served |
| Phone-keyboard text | Should allow | Typographic quotes and emoji | Served |
| Basket JSON with free-text notes | Should allow | "Ring the bell, don't hoot" delivery note | Served |
| Internal page from outside | Should deny | Same request from an unlisted network | Refused and logged |
| Known-bad samples | Should deny | The security team's maintained set | Refused and logged |
Should-allow cases are synthetic messages modelled on real shopping patterns, never stored shopper messages. Each confirmed false block becomes a permanent case, as in every incident becomes a test case. The suite runs in the pipeline against a staging WAF built from the same code as production.
Stage the rollout, watch the shoppers, keep the way back short
Detect first. Microsoft's Azure WAF guidance suggests detection mode while tuning, and notes it provides no protection. AWS recommends testing rules in count mode on production traffic before enabling them. CRS documentation describes another route: block from the start with very high anomaly-score thresholds, lowered gradually. Either way, exposure grows on evidence.
Block a slice. Block for a small share of traffic or one app version and compare it with the rest, as in a canary release.
Monitor blocked legitimate traffic. Ubuntu Grocer watches the block rate per journey step and app version, and especially blocks on signed-in shoppers with a recent completed order, who are very likely legitimate. A jump in "something went wrong" support contacts is a security signal too.
Make exceptions narrow and owned. CRS guidance is to tune with rule exclusions rather than editing rules. Choose the narrowest exclusion that works. An exception for one field on one path, with an owner and an expiry date, is a decision. Switching off a rule group to make a symptom disappear is a new false allow.
Rehearse rollback. Microsoft recommends holding WAF configuration and exclusions as code; that also keeps the previous version one change away. AWS documents that changes take from seconds to minutes to propagate, so old and new behaviour briefly coexist. Know how long your rollback really takes before Black Friday.
Who owns the trade-off
Tightening a rule trades protection against blocked shoppers, and neither security nor digital commerce should make that trade alone. At Ubuntu Grocer, Thandeka and Lindiwe Dube, head of digital commerce, sign off journey-affecting rule changes together, and the blocked-legitimate rate sits on both their dashboards. That is DevSecOps as a culture change: a shared number, not a scanner.
The Polokwane shopper never knew any of this. Her next month-end order went through first time.
Questions to take into your architecture review
- For each rule protecting the shopping journey, can you show both a should-allow test and a should-deny test?
- Which access rules are deny-lists, and is any internal page protected only by the WAF?
- Does your journey load test pass through the edge, with realistic conversation length and mixed-language text?
- When the edge blocks a request, can support join it to a shopper's complaint and a journey trace?
- Who sees the blocked-legitimate rate, and who is alerted when it moves?
- How do rule changes reach production: detect or count first, a partial slice, then full blocking?
- Are exclusions narrow, owned and dated, or are whole rule groups switched off?
- How long does a rollback take to apply everywhere, and when was it last rehearsed?
Key points
- WAF rules fail in two directions. False blocks look like AI faults; false allows go unnoticed.
- Deny-lists fail open; allow-lists fail closed. Read each rule aloud as a sentence and test it from outside.
- Conversational traffic looks like attack traffic: punctuation, non-ASCII text, encoded JSON and payloads that grow each turn.
- Roll rules out like releases: tests, detect first, a partial slice, full blocking, then monitoring of blocked legitimate shoppers.
- Keep exceptions narrow and owned, configuration as code, and rollback rehearsed.
CloudNala's point of view
Security controls in front of an AI journey are part of the journey and deserve the same engineering as the agent. In reviews of retail-agent programmes we often find careful evaluation of the model and none of the edge, so a quick rule change can undo months of conversion work. The fix is not weaker security. It is two-way tests, staged rollout and a blocked-legitimate number that security and commerce both own.
Next: Part 14, SRE agents need read-only by default. All parts are on the series page.
Background reading (checked on 5 November 2026): CRS paranoia levels, false positives and tuning and anomaly scoring; Azure WAF best practices; AWS WAF testing and tuning.
How CloudNala can help
CloudNala reviews the security controls in front of conversational journeys: we map which rules touch each journey step, build a two-way test corpus from realistic multilingual shopping traffic, check for fail-open conditions, and design a staged rollout and rollback path with a blocked-legitimate metric that security and digital commerce own together.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agentic Retail Review or write to us at consult@cloudnala.co.za