A KV cache handoff bug let GLM 5.3 responses carry other users' data

The most useful thing about this incident is where the failure sat. Nothing in the model weights leaked, and the model didn't misbehave in any way its designers would have predicted. A prompt longer than the model's KV cache pool was rejected by the scheduler, but the rejection never reached the decode node. Decode then generated a reply while reading memory pages that still held earlier requests' data. For a period between August 27 and September 9, 2026, that meant some GLM 5.3 responses on Workers AI contained text from other customers' sessions.

Cloudflare says a security researcher, Gadi Evron of Knostic, alerted it on September 8. The affected model was @cf/zai-org/glm-5.3. Cloudflare says no other model on Workers AI was affected, and it has restored GLM 5.3 with a corrected configuration and a patched engine. The company describes the leak as involving 155 responses, all to prompts longer than 681,280 tokens. After removing its own tests and the researcher's, it says the remaining requests looked like ordinary long-context traffic in both timing and volume, and it found no evidence of deliberate exploitation.

GLM is a text generation model from Z.ai aimed at agentic coding workloads. Its 1,048,576-token context window is the reason this bug mattered: most customers never send prompts anywhere near that size, so the failure path was rarely reached, and it was only reached by a small set of requests.

Two problems that only failed together

Cloudflare traces the exposure to two overlapping problems. The first was a configuration mismatch. The second was a bug in how SGLang, the open-source inference engine Cloudflare runs, handled over-length requests in disaggregated prefill/decode mode.

On the configuration side, the GLM 5.3 deployment advertised a 1,048,576-token context, but the KV cache pool on the prefill GPUs only held 681,280 tokens. Cloudflare says it knew about the gap before launch. Its pre-launch tests returned successful HTTP responses with streamed bodies and no error frames at long context, and the team concluded the engine must be handling the difference gracefully. It wasn't. Cloudflare's own summary of the lesson is blunt: it validated status codes and throughput at long context but never checked whether the model's output was coherent.

The engine bug is the more interesting technical story. In SGLang's single-node mode, an over-length request gets an error. The scheduler records the rejection, swaps the prompt for a one-token placeholder so the request can drain through the normal batch path without allocating real memory, and returns an error to the client. That design works when only one node is involved.

Disaggregated mode breaks the assumption. The admission queues on both the prefill and decode sides checked whether a request was finished, not whether it was rejected. A rejected request stored its status in a field the check didn't read, so it looked live. Prefill admitted it, ran a trivial forward pass over the placeholder token, and transferred one page of KV cache to decode. Decode had already reserved pages for the full prompt length, and it accepted the handoff.

What decode actually read

Here's the sequence Cloudflare lays out for a 8-page prompt, which generalizes to any over-length request:

1. Bootstrap. The router passes the prompt length to decode, which reserves eight KV pages.
2. Prefill admission. The rejected request passes the admission check because it isn't marked finished.
3. Prefill forward pass. Prefill runs on the one-token placeholder.
4. KV transfer. Prefill sends one page to decode and signals completion.
5. Decode admission. Decode accepts the handoff, since it uses the same flawed check.
6. Decode attention. Decode attends over all eight reserved pages. Only the first page was written for this request. The remaining seven still held prompts and partial generations from earlier requests.
7. Response. The client receives HTTP 200 instead of 400, with a prompt_tokens count that matches the long prompt and a coherent reply. The reply arrives faster than an eight-page prefill should take, which is the kind of detail that's easy to miss in a log.

The reply is coherent because the model is generating from real-looking context, and that context includes the other users' data. This is why the output looked healthy to the checks Cloudflare was running. Every request over the threshold followed this path, so the behavior was deterministic rather than a timing accident. Cloudflare says a bug report about the same behavior had been filed with SGLang as issue #30233 in July 2026, and that its team noticed it while investigating this incident.

The fix, in two layers

Cloudflare's immediate remediation had two parts, and it's worth keeping them separate because they fail differently.

The first change was to make the pool hold the full context. Cloudflare aligned the GLM 5.3 configuration with its GLM 5.2 setup, which freed enough GPU memory for the prefill KV pool to reach about 2.04 million tokens, roughly twice the context window. A prompt that fits the context window now always fits the pool, so the rejection path isn't reached for valid requests. That prevents the bug from being triggered, but it depends on the configuration staying correct.

That's why the second change matters more over time. Cloudflare patched the engine so that a request carrying a pending rejection is retired at the disaggregation admission queue, with its original error, before any bootstrap, allocation, or KV transfer happens. The company also had the decode node verify that the number of tokens transferred matches the number it reserved. Cloudflare proposed this upstream as #38704. A related fix, #36988, was already on SGLang's main branch for a neighboring failure path, and maintainers merged #38935, which rejects these requests earlier and follows Cloudflare's approach.

The reasoning behind the layering is worth stating plainly. Pool size is a derived value. SGLang sizes it at startup from whatever GPU memory is left after loading the weights, while the context length is a separate configuration value. Quantization changes, tensor parallelism changes, CUDA graph sizes, and a future model version can all shift the pool in ways that are easy to overlook. Any of those could push a deployment back into the failure path. The engine-side fix is the backstop for the day the configuration check is wrong.

Cloudflare validated both changes against the researcher's reproduction script and its own canary-marker tests before restoring access.

How the GLM 5.3 deployment is structured

The architecture explains why the bug crossed a node boundary at all. Workers AI runs GLM 5.3 in a disaggregated setup. A pool of prefill GPUs processes the prompt and produces its KV cache. That cache moves over remote direct memory access (RDMA), using Mooncake, to a pool of decode GPUs that generate the response.

The KV cache lives in GPU memory as a shared pool of fixed-size pages. Requests get pages when admitted and release them at completion. Pages aren't zeroed between requests. The next request that receives a page overwrites it. That's standard practice in inference engines because zeroing costs performance, but it places the burden on the engine to track which request owns each page at every moment.

Ownership gets complicated here. Within a single node, token generation and host-memory offload both touch the same memory with different lifetimes. Across nodes, prefill holds a simultaneous mutable reference to pages it has written to decode. Any mismatch in that shared picture is a potential read of someone else's data.

Why the testing missed it

Cloudflare's own write-up is candid about the gaps. The team saw HTTP success codes and a streaming body with no error frames during benchmarks. It assumed graceful handling. It had also observed that the pool was smaller than the context, so it wasn't ignorant of the mismatch, but it didn't treat it as a correctness risk. The company now checks KV pool size against context length directly from the running engine before any disaggregated deployment change ships, rather than inferring health from response codes.

That's the most transferable lesson here for anyone serving long-context models. Latency and status code checks can't detect a response that's fluent and wrong. Output validation has to include some check on what the model was actually conditioned on, not only whether it answered.

A class of bugs, not one

Cloudflare frames this incident as one instance of a wider category of KV cache memory-safety bugs it has been working on for several months. In each case, a page is freed, reused, or read while something else still holds a claim on it. The company sees two common causes.

The first is page lifetime inside a single node. Token generation, host offload, the KV transfer receiver, and the abort and error paths can all hold a page at once. If one path frees it while another is still reading, a new request can receive a page that's still in use. Cloudflare's earlier decode-offload and grammar-error fixes, #24226, #30899, and #30937, addressed this pattern.

The second is a race across nodes. If a request is aborted while prefill is still writing KV cache over RDMA, the writes already in flight can't be recalled. If decode frees those pages immediately, they can be handed to another request, and the late writes land in that request's cache. The result is corrupted output, or one user's data inside another user's response, with no error raised.

The incident had the same outcome as these races but a different mechanism. The rejected request went through the handoff every time, so it wasn't a timing problem. Decode was reading pages the request never wrote. None of the previously known races covered a case where prefill rejected a request and decode never learned of it.

What changed upstream

Cloudflare met with the SGLang maintainers at RadixArk on September 10, the day after restoring service. Both sides agreed that RadixArk would investigate the race conditions Cloudflare had reported, look at upstreaming an integrity-checking idea, and build reproduction tests for this class of bug.

Several changes have landed on SGLang's main branch. Most are in v0.5.22, and Cloudflare recommends operators either upgrade to that release or backport the patches. The main ones:

• Admission-level rejection (#38935, v0.5.21): over-length requests are rejected before bootstrap, allocation, or transfer.
• Router cancellation (#39461, #41612): sgl-router cancels both halves of a request on client disconnect or prefill failure.
• Deferred KV release (#37454, #41023): pages are held until transfer completion is confirmed, first enabled only on supported backends and then turned on by default.
• Per-transfer checksums (#39500, v0.5.21): the --disaggregation-enable-kv-checksum flag lets decode verify transferred KV before generating. A cleanup, #39729, is merged and awaiting release.
• Bounded transfer batches (#32758): an optional cap on Mooncake batch size, so one timed-out call can't leave an unbounded set of writes in flight.
• Failure-notification sharing (#36612): prefill-to-decode failure signaling is shared across Mooncake, NIXL, and MORI.
• Per-request abort tags (#37077, v0.5.22): a reused request ID can't consume a stale confirmation.
• Deferred-release metrics (#41128, v0.5.22): count, duration, and whether each hold ended on drain confirmation or timeout.

Cloudflare has also contributed proposed fixes and a reproduction script for the cross-node race, including #31357, #38961, and #32564. Its proposal in #38961 was superseded by a set of follow-up changes, and the work also includes a draft to zero KV pages on release, #46435, which is still a proposal.

What operators running SGLang should do

Cloudflare's guidance applies to anyone running SGLang in disaggregated prefill/decode mode, and it's concrete enough to act on this week.

1. Measure the KV pool against context length on every prefill and decode node. Read the value from the engine's startup log, and recheck after every change to weights, quantization, parallelism, or memory fraction. If the pool is smaller, lower the context length to fit. Don't rely on a proxy to enforce the limit, and don't assume the engine will reject what doesn't fit.
2. Upgrade to a build that includes #36988 and #38935, which means v0.5.22 or later. That makes over-length requests fail before handoff regardless of pool size.
3. Run deferred KV release on both roles and bound Mooncake transfer batches so a timed-out call can't leave a long tail of writes outstanding.
4. Turn on corruption detection where your model supports it. --disaggregation-enable-kv-checksum and --kv-canary don't yet cover every architecture. Confirm in your logs that requests are actually being verified. Cloudflare says it's working with SGLang to extend both checks to more models, including GLM 5.3, GLM 5.3 Flash, and DeepSeek V4.
5. Keep your load balancer from generating unnecessary aborts. Let requests proceed up to the first token, so a client disconnect doesn't trigger a cross-node cancellation mid-transfer.

Cloudflare also made changes to its own load balancer along these lines. It now holds the prefill connection until prefill finishes and the decode connection until the first token, then closes both. It added a deployment check covering pool size against context length, deferred release on both roles, transfer batch bounds inside the release timeout, and required patches.

Integrity by construction, and its limits

Configuration checks and lifecycle fixes shrink the number of ways a page can be misused. They can't make misuse impossible, and this incident is a reminder that the next bug will likely look different. The more durable approach is for the engine to detect, at read time, that a request is about to attend over a page it doesn't own, then fail that one request rather than return someone else's data.

Cloudflare built a prototype of this over the summer and described it in August. Each physical KV page carries a tag that changes on reallocation. Each request records the tags it expects, and the mapping is checked before decode reads the cache. In Cloudflare's tests, the overhead was under 1% in throughput and tail latency. The team tested it on GLM 5.2 in mid-August but disabled it shortly afterward because of reliability problems. The prototype was also tied to specific attention backends and model architectures, so it couldn't serve as a general fix.

The work now is to build a production version into the engine itself, one that works across models rather than as Cloudflare-specific patches. The per-transfer checksum in #39500 covers the handoff from prefill to decode. Page-level tags would go further, because they check every read, including reads long after the transfer. That distinction matters. A checksum can confirm the bytes arrived intact. It can't tell you whether the page belonged to the request that's reading it now.

Cloudflare is also pointing its formal verification work at this problem. It has previously used formal methods on distributed systems, including conflict-free IP selection for its authoritative DNS servers and consistency bugs found in a model of its Meerkat QuePaxa implementation. The plan now is to model and prove the safety of an RDMA-based KV cache transfer protocol, starting from production SGLang with Mooncake. The team expects gaps in what proofs can establish, and it intends to use the proven parts to focus review and monitoring on the remaining ones. For inference operators, that's a sign that memory safety in KV cache transfer is becoming an engineering discipline with its own tooling, not just a matter of careful patches.

Timeline

All times are UTC.

Credit and what comes next

Cloudflare credits Gadi Evron for a clear, reproducible script and for giving the company time to fix the problem and coordinate with SGLang before publication. It also credits the SGLang team at RadixArk, which treated the bug as its own problem and landed upstream fixes quickly.

The practical read for anyone serving long-context open models is this. The failure needed a configuration gap and an engine path that trusted the wrong status flag, and neither alone would have leaked data. Checking pool size against context length, upgrading to SGLang v0.5.22 or later, and enabling corruption checks where supported closes the known paths. Whether page-level integrity checks can become a default in the engine is the open question, and it's the one to watch as the SGLang work proceeds through the next few releases.