llm-d 1.0 Is About Day 2: Tenants, Upgrades and Agent Traffic

Serving a frontier model fast stopped being the hard part a while ago. Keeping it serving for many tenants, through upgrades, under load that arrives in bursts — that is where inference stacks show their seams, and it is what llm-d 1.0 was built for. The project now has a stable configuration API, priority- and fairness-aware flow control, coordinated in-place upgrades of disaggregated prefill/decode deployments, and session-aware routing for agent traffic that reuses enormous context. Under a year and a half old, with more than 500 contributors across 25-plus organizations, it has become the substrate several clouds and a few hundred engineering teams actually run. The interesting question for anyone self-hosting is no longer whether it is fast enough. It is what you inherit when you operate it.

Any model, any accelerator, any cloud was the 0.x promise. v1.0 is about operating it.

1. Day 1 versus Day 2, in concrete terms

The 0.x series had one goal, reach: inference that is not locked to one vendor, one cloud, one engine. It got there — NVIDIA, AMD and Intel GPUs plus Google TPUs, managed clouds and sovereign on-prem fleets, vLLM and SGLang, all behind one routing and scheduling layer with prefix-cache-aware routing, KV cache offloading, prefill/decode disaggregation and wide expert parallelism. Day 2 is the part nobody benchmarks. A production fleet carries mixed traffic: latency-sensitive interactive requests next to bursty agent workloads that spawn parallel work without warning. Priorities compete and something has to give. Upgrades cannot mean downtime. Disaggregated deployments multiply the number of moving parts that must roll forward together. And underneath all of it, inference has to run like a service, with somebody on call.

# The v1.0 routing policy surface. Field names follow the v1 config reference;
# the point of the snippet is the shape of the decision, not the spelling.
# Defaults are first-come-first-served, so nothing here applies until you opt in.
flowControl:
  enabled: true
  # One flow = fairness ID (tenant or workload) x priority band.
  flows:
    - id: tenant:payments
      priority: premium
      # Premium keeps a reserved share under saturation.
      minShare: 0.30
    - id: tenant:internal-tools
      priority: best-effort
    - id: workload:nightly-batch
      priority: batch
  # Shed load probabilistically instead of failing hard when capacity runs out.
  admission:
    mode: probabilistic
    # Start conservative, then raise it while watching p99 for the premium band.
    maxUtilisation: 0.85

# Two things this file does not do for you:
#   1. It does not protect a tier you never declared.
#   2. It cannot be validated by a single-tenant load test. Fairness only shows
#      up when two tenants compete, which is what the next snippet measures.

2. Flow control: who gets the GPU when demand exceeds capacity

1.0 makes multi-tenant behaviour a first-class property of the router instead of something operators approximate with separate fleets. Every request is classified by a flow key that combines a fairness ID — a tenant or a workload — with a priority band. The router keeps a separate queue per flow and dispatches by priority first, then cycles fairly across tenants inside the same band. Under saturation it does no-regret scheduling: it holds requests in the router rather than committing them to backend queues that are already overloaded, so one tenant's burst does not blow up everyone else's latency. Per-workload policies are expressible, and a probabilistic admission controller sheds load gracefully instead of failing hard. All of it sits under a single flowControl configuration surface, and the defaults still mimic first-come-first-served, so existing deployments behave exactly as before until somebody opts in.

# Verify the fairness claim yourself. Two tenants, one premium and one that
# bursts, and separate latency distributions per tenant.
import asyncio, random, statistics, time
import httpx

ENDPOINT = "http://llm-d-router.default.svc/v1/completions"
PROMPT = "Summarise the following incident report in two sentences. " * 12


async def one(client, tenant, priority, results):
    t0 = time.perf_counter()
    try:
        r = await client.post(ENDPOINT, json={
            "model": "open-model", "prompt": PROMPT, "max_tokens": 64,
        }, headers={"x-tenant-id": tenant, "x-priority": priority}, timeout=60)
        r.raise_for_status()
        results[tenant].append(time.perf_counter() - t0)
    except Exception as exc:
        results[tenant + "_errors"].append(type(exc).__name__)


async def main():
    results = {"premium": [], "burst": [], "premium_errors": [], "burst_errors": []}
    async with httpx.AsyncClient() as client:
        tasks = []
        # 10 premium users, steady; 60 burst requests fired in one go.
        for _ in range(10):
            tasks.append(one(client, "premium", "premium", results))
        for _ in range(60):
            tasks.append(one(client, "burst", "best-effort", results))
            if random.random() < 0.2:
                tasks.append(one(client, "premium", "premium", results))
        await asyncio.gather(*tasks)

    for tenant in ("premium", "burst"):
        lat = sorted(results[tenant])
        if not lat:
            print(tenant, "no samples; check the error bucket")
            continue
        p50 = statistics.median(lat)
        p95 = lat[int(len(lat) * 0.95) - 1]
        print(f"{tenant:<8} n={len(lat):<3} p50={p50:.2f}s p95={p95:.2f}s "
              f"errors={len(results[tenant + '_errors'])}")

asyncio.run(main())

# What you want to see: the premium p95 barely moves when the burst arrives.
# If both distributions track each other, the policy is not in effect — check
# that the router actually loaded it, and that the header names match.

3. DisaggregatedSet: upgrading prefill and decode without a parallel fleet

Disaggregated serving splits prefill from decode across pools, which is great for efficiency and terrible for change management: the pair has to move together, and a half-rolled-out pair serves wrong results rather than slow ones. Two pieces address that. Router high availability keeps a hot standby of the Endpoint Picker ready to take over scheduling if the active instance is replaced, so routing never goes dark mid-rollout. And DisaggregatedSet manages prefill and decode replicas as one logical unit: readiness is gated on a snapshot of the underlying pods, and routing is revision-aware, so a request only reaches a prefill/decode pair once both sides are fully rolled out and compatible. Coordinated lockstep upgrades then become safe to run in place. On the request path, a coordination layer consolidated into llm-d-router drives each request through its encode, prefill and decode stages, which turns a multi-phase pipeline into one flow instead of something the caller stitches together.

#!/usr/bin/env bash
# Watch a rollout from the outside. Run this in one terminal, roll the
# DisaggregatedSet in another, and read the error count at the end.
set -uo pipefail

URL="${1:?usage: rollout_watch.sh http://llm-d-router.default.svc/healthz}"
DURATION="${2:-600}"
INTERVAL=0.5

ok=0; bad=0; slow=0
end=$(( $(date +%s) + DURATION ))

while [ "$(date +%s)" -lt "$end" ]; do
  start=$(python3 -c 'import time;print(time.time())')
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$URL" || echo 000)
  elapsed=$(python3 -c "import time;print(round(time.time()-$start,2))")

  if [ "$code" = "200" ]; then
    ok=$((ok+1))
    # Slow but successful is still a client-visible outage during a rollout.
    awk -v e="$elapsed" 'BEGIN{exit !(e > 2.0)}' && slow=$((slow+1))
  else
    bad=$((bad+1))
    echo "$(date +%H:%M:%S) $code ${elapsed}s"
  fi
  sleep "$INTERVAL"
done

echo "ok=$ok errors=$bad slow_over_2s=$slow"
[ "$bad" -eq 0 ] || exit 1

# Any non-zero error count here means the revision-aware gating is not doing
# its job: a request reached a prefill/decode pair before both sides were ready.

4. Agent-aware serving: the cache is the workload

Agentic traffic has an unusual shape: most of the compute goes to handling context rather than producing the final tokens, and the same context comes back turn after turn. 1.0 serves it natively. Session-aware routing keeps a multi-turn conversation pinned to the replica that already holds its KV cache; a session-state-producer plugin tags requests with session state and the router sends follow-ups back to the same backend. Affinity is sticky until saturated, then rebalances — trading a little cache locality for headroom instead of dropping requests. Turn-priority scheduling lets the router reason about where a request sits inside an ongoing exchange, so an active agent loop stays responsive instead of every turn being treated as an isolated call. Native Responses API state, including previous_response_id, lets the server carry context across turns so the agent does not resend its whole history. For anyone building agents, the practical effect is lower latency and lower cost without changing application code.

# Does session affinity actually pay? Ask the server twice with the same
# conversation and once with a cold one, and compare second-turn latency.
import time, uuid
import httpx

BASE = "http://llm-d-router.default.svc/v1"
LONG_CONTEXT = "Line item %d: quantity, unit price, tax code.\n"
CONTEXT = "".join(LONG_CONTEXT % i for i in range(400))

client = httpx.Client(base_url=BASE, timeout=120)


def turn(payload):
    t0 = time.perf_counter()
    r = client.post("/responses", json=payload)
    r.raise_for_status()
    return time.perf_counter() - t0


def warm_conversation():
    first = client.post("/responses", json={
        "model": "open-model",
        "input": "Here is a price list:\n" + CONTEXT + "\nReply with OK.",
    }).json()
    # The id is what lets the server carry context instead of you resending it.
    return first.get("id")


# Warm session: second turn reuses the KV cache pinned to that replica.
sid = warm_conversation()
warm = turn({"model": "open-model", "previous_response_id": sid,
             "input": "Now give me the total number of line items."})

# Cold session: same question, no history to lean on.
cold = turn({"model": "open-model",
             "input": "Here is a price list:\n" + CONTEXT +
                      "\nGive me the total number of line items."})

print(f"warm second turn: {warm:.2f}s")
print(f"cold equivalent:  {cold:.2f}s")
print(f"ratio: {cold / warm:.1f}x")

# On agent traffic this ratio is the whole argument for session-aware routing.
# If it lands near 1.0, either the plugin is not tagging requests or the
# conversation is too short for the cache to matter.

5. The numbers people will quote, and where they came from

The performance case was already made before 1.0. Google Cloud builds the GKE Inference Gateway on the llm-d Endpoint Picker and reports time-to-first-token down by more than 35% in Vertex AI production while prefix-cache hit rate doubled from 35% to 70%. AWS runs disaggregated inference on SageMaker HyperPod with llm-d and reports up to 70% more tokens per second than standard vLLM. Tesla and Red Hat report 3x output tokens per second and half the time-to-first-token on Llama 3.1 70B across AMD MI300X. IBM Research self-hosts frontier open models for coding agents and reports up to 3,000 concurrent agents on 544 H100s at roughly a 98% prefix-cache hit rate, about $0.11 per million tokens, which they put at around 11.5x cheaper than a proprietary stack. DigitalOcean serves a trillion-parameter model across mixed AMD and NVIDIA hardware, JetBrains runs its Junie agent behind it, and Mistral uses it for regional inference. Every one of those is a vendor-published number, which means the right move is to reproduce the shape on your own traffic before quoting it.

# Self-host or call an API? The comparison turns on prefix reuse, so model it
# instead of arguing about it. All inputs are yours; only the arithmetic is fixed.
from dataclasses import dataclass


@dataclass
class Scenario:
    name: str
    requests_per_day: int
    prompt_tokens: int          # average prompt size
    output_tokens: int          # average completion size
    prefix_hit_rate: float      # measured, not assumed
    gpu_hour_usd: float         # your actual rate, including idle headroom
    gpus: int
    tokens_per_sec: float       # measured at your concurrency
    api_in_usd: float           # per million input tokens
    api_out_usd: float          # per million output tokens


def self_host(sc: Scenario, days: int = 30) -> float:
    return days * 24 * sc.gpus * sc.gpu_hour_usd


def api(sc: Scenario, days: int = 30) -> float:
    reqs = sc.requests_per_day * days
    # Cached prefix tokens are usually billed at a discount; the point of the
    # model is to make that assumption explicit, not to bury it.
    billed_in = reqs * sc.prompt_tokens * (1 - sc.prefix_hit_rate * 0.9)
    billed_out = reqs * sc.output_tokens
    return (billed_in / 1e6) * sc.api_in_usd + (billed_out / 1e6) * sc.api_out_usd


def utilisation(sc: Scenario, days: int = 30) -> float:
    reqs = sc.requests_per_day * days
    work = reqs * (sc.prompt_tokens + sc.output_tokens)
    capacity = sc.gpus * sc.tokens_per_sec * days * 24 * 3600
    return work / capacity


if __name__ == "__main__":
    sc = Scenario(
        name="coding-agents", requests_per_day=40_000, prompt_tokens=18_000,
        output_tokens=900, prefix_hit_rate=0.6, gpu_hour_usd=2.5, gpus=8,
        tokens_per_sec=2_500, api_in_usd=1.0, api_out_usd=4.0,
    )
    print(f"{sc.name}: self-host ${self_host(sc):,.0f}/mo  api ${api(sc):,.0f}/mo "
          f"util {utilisation(sc):.0%}")

    # Read the utilisation column first. Below roughly a third, the GPUs you are
    # paying for are mostly idle, and no routing improvement fixes that.
    # Above it, prefix reuse at 60% plus flat per-token pricing is what makes
    # the self-hosted number stop being a rounding error.

6. Before you adopt it, decide what you are signing up to operate

A control plane that gives you tenants, priorities and zero-downtime upgrades also gives you the job of running them. The defaults are deliberately backward compatible, which means an untouched install will not protect a premium tier — you have to write the policy and test it under saturation, and a load test with one tenant proves nothing about fairness. Two things are worth doing before a migration: measure your real prefix reuse rate, because every cost argument here depends on it, and measure what fraction of your traffic is multi-turn, because session affinity only pays for itself when conversations actually come back. If neither number is high, a managed endpoint is still the cheaper answer, and choosing it is not a failure of ambition.