When the KV Cache Pool Is Smaller Than the Context Window, Other People's Prompts Come Back

On October 26, 2026, Cloudflare and LMSYS published coordinated write-ups of something that is rarely documented in public: a tenant isolation failure inside an inference engine. GLM-5.3 on Workers AI ( @cf/zai-org/glm-5.3 ) shipped on August 27 with a prefill KV cache pool smaller than its own context window, and until September 9 a small number of customers received responses containing data from other customers' sessions. 155 responses were affected, every one of them to a prompt longer than 681,280 tokens. The interesting part is not that a cloud had an incident. It is the structural detail underneath: in a disaggregated inference deployment, nothing forces the KV cache pool size to agree with the context length the model advertises. That check belongs to the operator, not the engine.

The leak needed no exploit: a rejected request still walked through the prefill-to-decode handoff.

1. Background: a pool of pages that is never zeroed

Cloudflare serves GLM-5.3 with SGLang in a disaggregated setup: a pool of prefill GPUs processes the prompt and produces the KV cache, then hands it over RDMA via Mooncake to a pool of decode GPUs that generate the response. The KV cache is a shared pool of fixed-size pages in GPU memory. A request is assigned pages when admitted and releases them when it finishes, and pages are not zeroed between requests — the next request simply overwrites them. That is a performance necessity, and it makes ownership tracking the engine's job. Ownership is harder than it sounds: within a node, token generation, offload to host memory, the transfer receiver, and the abort and error paths can all hold the same page at once; across nodes, the remote prefill node keeps a simultaneous mutable reference. And the pool size is not a configuration value. SGLang sizes it at startup from whatever GPU memory remains after the weights load, while the context length is a separate setting. Nothing forces the two numbers to agree.

# The check Cloudflare now runs before any disaggregated deployment change.
# SGLang does NOT force the KV pool size to match the context length,
# so read both numbers from the running engine and compare them.

# 1) startup log prints the pool size it sized from leftover GPU memory
kubectl -n workers-ai logs deploy/glm-prefill | grep -iE "kv.?cache|pool|max_total_tokens"
#   -> max_total_num_tokens: 681280

# 2) context length is a separate config value you set yourself
curl -s localhost:30000/get_model_info | jq '.max_context_length'
#   -> 1048576   <-- 681280 < 1048576: this deployment is unsafe

# 3) the metric mirror of the same number, for alerting
curl -s localhost:30000/metrics | grep sglang:max_total_num_tokens

# Rule: pool >= context, on BOTH prefill and decode nodes,
# re-checked after every change to weights, quantization,
# tensor parallelism or memory fraction.

2. Two problems had to overlap before anything leaked

Problem one was configuration. GLM-5.3 was deployed with a 1,048,576-token context window while the prefill pool held 681,280 tokens. Cloudflare knew about the mismatch before launch and expected the engine to return errors; long-context tests came back with successful responses, so it was not treated as a defect. Problem two was how SGLang handled the resulting over-length requests in disaggregated mode. A prompt that did not fit was marked rejected, its prompt swapped for a one-token placeholder so the request could drain through normal batch machinery. On a single node that is safe: the placeholder runs, the batch finishes, the client gets an error. In disaggregated mode, both admission queues checked req.finished(), and a rejected request stores its rejection in to_finish rather than finished_reason — so finished() still returned False and the request reported itself live. From there the handoff ran on rails. Decode reserved eight KV pages for a prompt it was told was eight pages long. Prefill ran a trivial forward pass on one token and transferred one page, then signalled completion. Decode read all eight, and pages one through seven still held other users' prompts and partial generations. The client received HTTP 200, prompt_tokens matched the long prompt, and the reply was coherent.

# A canary-marker test: does a reply ever carry bytes from
# another request? Cheap to run, and it fails loudly.
import uuid, requests

MARKERS = {}

def ask(prompt_tokens: int, session: str) -> str:
    marker = f"CANARY-{session}-{uuid.uuid4().hex[:8]}"
    MARKERS[marker] = session
    body = {"model": "glm-5.3", "max_tokens": 16,
            "prompt": marker + " x" * prompt_tokens}
    return requests.post("http://prefill:30000/generate", json=body, timeout=600).text

# Long request that the pool cannot hold, then a short one right behind it.
ask(700_000, "big")
out = ask(64, "small")

for marker, owner in MARKERS.items():
    if marker in out and owner != "small":
        raise SystemExit(f"LEAK: response for 'small' carried {owner}'s marker")

3. What they changed before turning the model back on

The timeline is worth copying into your own incident process. Report received September 8 at 23:11 UTC. Verification began the next morning. Incident declared at 13:22, and product security reproduced the leak with two independent accounts within 25 minutes. At 14:00 the model went private and requests returned HTTP 400. Root cause was identified at 14:35 as the prefill pool being smaller than the context length, and a canary node with a larger pool was confirmed clean. All prefill nodes were rolled by 14:52. At 19:04 the full deployment was rebuilt with the corrected configuration plus a Cloudflare-authored engine patch, validated against the researcher's reproduction script. Service restored at 19:40. Both fixes are worth internalising. First, make the pool hold the full context: the pool now holds about 2.04 million tokens, roughly twice the context length, so any prompt that fits the context fits the pool and the rejection path is never reached for a valid request. Second, do not rely on pool size, because quantization, tensor parallelism, CUDA graph sizes or a new model version all move it silently. Patch the engine so a request carrying a pending rejection is retired at the disaggregation admission queue, with its original error, before bootstrap, allocation or KV transfer. That went upstream as #38704, with SGLang landing #36988 in v0.5.20 and #38935 in v0.5.21.

The KV pool size is printed at startup and exposed as a metric. That is the number to compare against your context length.

# The rejection path, before and after. The bug is not memory maths,
# it is which flag the admission queues looked at.

# BEFORE (SGLang < v0.5.20, disaggregated prefill/decode)
def set_finish_with_abort(req, err):
    req.to_finish = err            # error recorded here ...
    req.prompt = PLACEHOLDER_1TOK  # ... prompt swapped for 1 token
    # req.finished() still returns False: it reads finished_reason,
    # not to_finish. So both admission queues admit the request.

def admit(req):
    return not req.finished()      # rejected request reports itself live

# AFTER (v0.5.20 / #36988, v0.5.21 / #38935)
# 1. rejected request is retired at the disaggregation admission queue,
#    with its original error, before bootstrap, allocation or KV transfer
# 2. decode verifies transferred token count == reserved page count
# 3. #38935 rejects at admission, so the handoff never starts

4. The testing blind spot is the part to remember

The most useful sentence in the report is an admission: during pre-launch testing Cloudflare validated HTTP status codes and engine performance at long contexts, but not model output and coherence. They were measuring "no error was raised" rather than "the answer is right." That trap is not specific to SGLang. A provider that returns 200 with another tenant's tokens passes every status-code check you own. Their new rule is blunt: compare KV pool size against context length, read directly from the running engine, before any disaggregated deployment change ships — no inference from response codes. Around that they added hardening that is mostly about ownership under failure: deferred KV release extended to prefill's source pages and to every failure type, not just aborts; bounded Mooncake transfer batches so a single timed-out call cannot leave an unbounded tail of writes; a load balancer that no longer aborts a request mid-transfer when the client disconnects, holding the prefill connection until prefill finishes and the decode connection until the first token; and corruption detection via --disaggregation-enable-kv-checksum plus the KV canary.

# Pre-flight gate for a disaggregated SGLang rollout.
# Four conditions; fail the deploy if any is missing.
set -euo pipefail

POOL=$(sglang_pool_tokens prefill)      # from startup log or metric
CTX=$(sglang_context_length prefill)
[ "$POOL" -ge "$CTX" ] || { echo "pool $POOL < context $CTX"; exit 1; }

# deferred KV release must be on for BOTH roles
sglang_flag_on prefill deferred_release
sglang_flag_on decode  deferred_release

# bounded transfer batches so one timed-out call cannot leave
# an unbounded tail of writes in flight
sglang_flag_le mooncake_transfer_batch_max 64

# corruption detection where the architecture supports it
sglang_flag_on decode kv_checksum  # --disaggregation-enable-kv-checksum
echo "pre-flight ok: pool=$POOL ctx=$CTX"

5. Four checks SGLang operators can run today

First, measure. On every prefill and decode node, read the KV pool token count from the startup log or the metric and compare it with your context length. If the pool is smaller, lower the context length. Do not assume a proxy enforces the limit and do not assume the engine rejects what does not fit — that assumption is precisely what failed here. Second, upgrade to v0.5.22 or later, which contains #36988 and #38935: the engine-side backstop for the day your first check is wrong. Third, run deferred release on both roles and bound transfer batches. Fourth, enable corruption detection where your architecture supports it, and check the logs to confirm requests are actually being verified — the checksum and canary flags do not yet cover every model, including GLM 5.3 itself. Cloudflare also names the durable answer: an engine that detects at read time that a request is about to attend over a page it does not own, and fails that one request instead of returning someone else's data. Their prototype tags every physical page with a value that changes on reallocation, records the tags each request expects, and validates the mapping before decode reads. In their tests the check cost under 1%.

Cloudflare rebuilt the deployment with a corrected config plus an engine patch before restoring the model.

# Where this class of bug goes next: check ownership at read time.
# Cloudflare prototyped page tagging over the summer; in their tests
# the check cost under 1% of throughput.

def decode_attention(req, pages):
    for page in pages:
        # every physical KV page carries a tag that changes when it
        # is reallocated; the request records the tags it expects
        if page.tag != req.expected_tags[page.index]:
            raise KvOwnershipError(
                page=page.index,
                expected=req.expected_tags[page.index],
                found=page.tag,
            )   # fail THIS request, never return someone else's data
        attn(page)

# Config checks and lifetime fixes shrink the set of ways a page
# can be misused. They do not make misuse impossible.