At half past five on a month-end Friday, the autoscaler does exactly what it was configured to do. Traffic to Ubuntu Grocer's meal assistant is climbing, so it adds orchestrator pods. Each new pod reports ready within seconds. Shoppers routed to them wait, then time out. A few minutes later the operational database starts refusing new connections, and journeys that were fine on the old pods begin to fail too. A shopper in Mthatha who asked for "samp and beans with something for the braai, for six" gets an error, twice.
Nothing in the demo, which ran on two pods, predicted any of this.
Ubuntu Grocer is a composite: a fictional South African grocery retailer whose scenario draws on lessons from real enterprise retail-agent delivery. Its people, systems and figures are illustrative and do not describe any single organisation.
Part 11 gave the team a trace for every basket. This part is about what those traces kept pointing at: not the models but the plumbing, invisible in a demo and decisive in production.
Readiness probes that lie
Kubernetes has three kinds of probe, and they do different things. The Kubernetes documentation is precise about the difference:
- A startup probe checks whether the application has finished starting. While one is configured and has not yet succeeded, the kubelet does not run the liveness or readiness probes. If it fails, the container is killed and restarted under its restart policy.
- A readiness probe decides whether the container should receive traffic. When it fails, the pod is removed from the endpoints of the Services that select it. The container is not restarted, and readiness probes run for the container's whole lifecycle.
- A liveness probe decides whether to restart the container, for example when it is running but unable to make progress.
Ubuntu Grocer's orchestrator had one readiness endpoint that returned 200 as soon as the web server was listening. That was true and useless: the pod had not yet opened its database pool, built its gateway client, loaded its configuration or warmed its catalogue lookup cache.
The failure compounds: cold pods are slow, so the autoscaler adds more cold pods, also marked ready. The probe told the truth about the process and a lie about the service.
The fix has three parts. Readiness gates traffic, so it checks what this pod needs to serve a shopper: configuration loaded, pool open, clients built, cache warmed. A startup probe defers the liveness and readiness checks while a slow starter warms up, so it is not killed halfway through. Liveness checks only that the process can make progress, never a shared dependency, because a liveness check on the database turns a database slowdown into a restart storm.
There is a trap on the other side. If every pod's readiness probe checks the model gateway and the gateway is rate-limited, every pod fails readiness at once, the Service has no endpoints, and the rules-only and plain-search rungs from Part 10 disappear with it. A shared dependency outage should degrade the journey inside the application, not empty the load balancer.
Pods times pool size
Each pod opens its own pool of database connections; the database has one limit for all of them. PostgreSQL, for example, defaults max_connections to typically 100, with three reserved for superusers, and the value can only be changed at server start. Nobody multiplies the two until the autoscaler does.
In our worked example, with illustrative numbers, the database is configured for 400 connections and 30 are held back for superusers, migrations and administration, so 370 are usable. The orchestrator has a pool of 20 per pod and scales from 10 to 24 pods. A background worker runs four pods with a pool of 10.
The demo used 40. A normal Tuesday uses 240. At month-end peak, 24 orchestrator pods want 480 and the worker another 40: 520 against 370. Then someone runs a rolling deploy during the peak. A Kubernetes Deployment's default maxSurge is 25%, so for a while 30 orchestrator pods and five worker pods are running, and demand reaches 650.
The arithmetic to do in design is short:
the sum, over every service using the database, of (maximum replicas × surge factor × pool size) ≤ max connections − reserved
Here, the worker at surge needs 5 × 10 = 50, which leaves 320 for 30 orchestrator pods: a pool of 10, for a total of 350.
Is 10 enough? Pool size follows how long each request holds a connection, not how many shoppers are online. If a pod handles, say, 40 journey steps a second and each holds a connection for 50 milliseconds, it needs about two connections on average. Pieter van Wyk's review found the real problem: one code path opened a transaction, called the model for the plan and wrote the result before committing. If every planning step took that path, holding a connection for a four-second model call would turn two connections into 160: the pool of 10 exhausts and requests queue behind it. Never hold a database connection across a model call. Read, release, call the model, then write.
A pooler such as PgBouncer can multiplex many client connections onto fewer server connections; its documentation notes that transaction pooling breaks a few session-based PostgreSQL features, so test first.
One client per process, not per request
Ruan Botha's team found an adapter that created a new AI gateway client, with a fresh credential lookup, for every model call: a new connection and Transport Layer Security (TLS) handshake each time. Microsoft's HttpClient guidelines for .NET explain why this is dangerous: closed connections do not release their ports immediately, and at high request rates the operating system can run out of available ports. Go's net/http documentation says the same in one line: clients hold cached connections, so reuse them rather than creating them as needed.
Long-lived clients have their own trap. The same Microsoft page notes that a client resolves DNS only when it opens a connection, so without a connection lifetime (PooledConnectionLifetime in .NET) it keeps talking to the old gateway address, which quietly undermines the failover designed in Part 10. Build clients once per process, reuse their connection pools, cache credentials until they near expiry, and make readiness wait until they exist.
The cache is a capacity decision
Redis, a common cache, defaults maxmemory to zero on 64-bit systems, which means no limit: it grows until the container hits its memory limit, at which point the kernel may kill it. Set a limit and choose the eviction policy deliberately: with noeviction writes fail when memory is full; with allkeys-lru the least recently used keys go first.
Ubuntu Grocer had a subtler problem: conversation state and the catalogue lookup cache lived in the same instance. Under pressure, eviction removed half-built plans as readily as product lookups. Redis's own guidance suggests separate instances for cache data and keys you cannot afford to lose. Session state is not cache.
Configuration drift is architecture
The last finding was the least glamorous. In staging the pool size was set explicitly; in production the variable was missing and a framework default applied. Probe timings had been copied from a service that starts in two seconds, so the failure threshold was too short for the orchestrator's warm-up, and new pods were restarted in a loop during scale-out.
Treat configuration as architecture: keep it as code, diff the effective configuration between environments before release, and log non-secret effective values at start-up so a trace can be read against what the pod actually ran. A passed test is not proof of production readiness for the same reason: the test ran against a configuration production did not have.
Questions to take into your architecture review
- What does each service's readiness probe check, and does any liveness probe depend on a shared dependency?
- Across every service using each database, does maximum replicas × surge factor × pool size fit the usable connection limit?
- Does any code path hold a database connection or transaction across a model call?
- Are HTTP and model clients created once per process, and are credentials cached?
- What memory limit and eviction policy does each cache instance have, and is session state separated from cached data?
- How do you detect configuration differences between staging and production before release?
Key points
- A readiness probe should say whether this pod can serve a shopper, not whether its process is listening. Use a startup probe for warm-up and keep liveness free of shared dependencies.
- Database demand is pods × pool size, at maximum replicas and during a rolling surge. Size the pool from that worst case.
- Pool size follows connection hold time. Never hold a connection across a model call.
- Reuse clients with a connection lifetime, set cache limits deliberately, and keep session state out of the cache.
- Configuration is architecture: diff it between environments before every release.
CloudNala's point of view
When we review agentic systems that struggled at launch, the root cause is rarely the model and often one of these five. Nobody demonstrates a connection pool to a steering committee. We would put the connection arithmetic, the probe definitions and the configuration diff on the same go-live checklist as the evaluation results, because they decide whether the shopper ever sees those results.
Next: Part 13, When security controls break the customer journey. All parts are on the series page.
How CloudNala can help
CloudNala reviews the hidden architecture behind AI journeys before peak trading: probes, connection arithmetic across every service and scaling limit, connections held across model calls, client lifetimes, cache limits and configuration drift. Kubernetes is the engine, not the platform covers the wider platform.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agentic Retail Review or write to us at consult@cloudnala.co.za