llm-d 1.0 真正解决的是「第二天」：多租户、升级和 Agent 流量

把前沿模型serve得快，早就不再是难题了。难的是让它在多租户下、跨越升级、在突发负载里一直serve得住——推理栈的裂缝就是在这一步露出来的，而 llm-d 1.0 就是冲这件事做的。这个项目现在有了稳定的配置 API、按优先级与公平性调度的流控、prefill/decode 分离部署的原地协同升级，以及面向「反复复用巨大上下文」的 Agent 流量的会话感知路由。不到一年半，500 多位贡献者、25 个以上组织，它已经成了好几家云和几百个工程团队实际在跑的底座。对任何自建推理的人来说，值得问的问题已经不是「它够不够快」，而是「把它跑起来之后，我要继承什么」。

「任何模型、任何加速器、任何云」是 0.x 的承诺；v1.0 讲的是怎么把它跑起来。

一、「第一天」和「第二天」，具体差在哪

0.x 系列只有一个目标：覆盖。推理不能被锁死在一家厂商、一朵云、一个引擎上。这件事它做到了——NVIDIA、AMD、Intel 的 GPU 加 Google TPU，托管云与主权本地集群，vLLM 与 SGLang，全部落在同一个路由与调度层后面，带前缀缓存感知路由、KV 缓存下沉、prefill/decode 分离和宽专家并行。而「第二天」是没人做基准测试的那部分。生产集群跑的是混合流量：对延迟敏感的交互请求，旁边就是会毫无预警地并行开花的 Agent 负载。优先级互相争抢，总得有人让步。升级不能等于停机。分离部署把「必须一起滚动前进」的部件数量成倍放大。而在这一切之下，推理得当成一项服务来运营，得有人在值班。

# The v1.0 routing policy surface. Field names follow the v1 config reference;
# the point of the snippet is the shape of the decision, not the spelling.
# Defaults are first-come-first-served, so nothing here applies until you opt in.
flowControl:
  enabled: true
  # One flow = fairness ID (tenant or workload) x priority band.
  flows:
    - id: tenant:payments
      priority: premium
      # Premium keeps a reserved share under saturation.
      minShare: 0.30
    - id: tenant:internal-tools
      priority: best-effort
    - id: workload:nightly-batch
      priority: batch
  # Shed load probabilistically instead of failing hard when capacity runs out.
  admission:
    mode: probabilistic
    # Start conservative, then raise it while watching p99 for the premium band.
    maxUtilisation: 0.85

# Two things this file does not do for you:
#   1. It does not protect a tier you never declared.
#   2. It cannot be validated by a single-tenant load test. Fairness only shows
#      up when two tenants compete, which is what the next snippet measures.

二、流控：需求超过容量时，GPU 给谁

1.0 把多租户行为做成了路由器的第一等属性，而不是让运维拿一堆独立集群去凑。每个请求按一个流键分类，流键由公平性 ID（租户或业务负载）和优先级带组合而成。路由器为每条流维护独立队列，先按优先级派发，同一优先级带内再在租户之间公平轮转。饱和时它做的是「无悔调度」：把请求留在路由器里，而不是塞进已经过载的后端队列，于是某个租户的突发不会把别人的延迟一起点着。策略可以按业务负载表达，还有一个概率式准入控制器，在需求超过容量时优雅削峰，而不是硬失败。这些东西都收在单一的 flowControl 配置面下，而且默认行为仍然等同先来先服务——只要没人主动开启，既有部署的表现一点不变。

# Verify the fairness claim yourself. Two tenants, one premium and one that
# bursts, and separate latency distributions per tenant.
import asyncio, random, statistics, time
import httpx

ENDPOINT = "http://llm-d-router.default.svc/v1/completions"
PROMPT = "Summarise the following incident report in two sentences. " * 12


async def one(client, tenant, priority, results):
    t0 = time.perf_counter()
    try:
        r = await client.post(ENDPOINT, json={
            "model": "open-model", "prompt": PROMPT, "max_tokens": 64,
        }, headers={"x-tenant-id": tenant, "x-priority": priority}, timeout=60)
        r.raise_for_status()
        results[tenant].append(time.perf_counter() - t0)
    except Exception as exc:
        results[tenant + "_errors"].append(type(exc).__name__)


async def main():
    results = {"premium": [], "burst": [], "premium_errors": [], "burst_errors": []}
    async with httpx.AsyncClient() as client:
        tasks = []
        # 10 premium users, steady; 60 burst requests fired in one go.
        for _ in range(10):
            tasks.append(one(client, "premium", "premium", results))
        for _ in range(60):
            tasks.append(one(client, "burst", "best-effort", results))
            if random.random() < 0.2:
                tasks.append(one(client, "premium", "premium", results))
        await asyncio.gather(*tasks)

    for tenant in ("premium", "burst"):
        lat = sorted(results[tenant])
        if not lat:
            print(tenant, "no samples; check the error bucket")
            continue
        p50 = statistics.median(lat)
        p95 = lat[int(len(lat) * 0.95) - 1]
        print(f"{tenant:<8} n={len(lat):<3} p50={p50:.2f}s p95={p95:.2f}s "
              f"errors={len(results[tenant + '_errors'])}")

asyncio.run(main())

# What you want to see: the premium p95 barely moves when the burst arrives.
# If both distributions track each other, the policy is not in effect — check
# that the router actually loaded it, and that the header names match.

三、DisaggregatedSet：不用另起一套集群，也能升级 prefill 与 decode

分离式服务把 prefill 和 decode 拆到不同池子里，对效率极好，对变更管理极差：这一对必须一起动，而「只滚了一半」的组合不会变慢，它会给出错的结果。两处设计针对这点。路由高可用会跑一个热备 Endpoint Picker，活动实例被替换时它接手调度，于是滚动过程中路由不会中断。DisaggregatedSet 则把 prefill 与 decode 的副本当成一个逻辑单元管理：就绪状态以底层 Pod 的快照为准，路由带版本感知，所以一个请求只有在两侧都已完全滚动到位且版本兼容时，才会被送到这对 prefill/decode 上。协同的同步升级因此可以原地做。在请求路径上，这一版把协调层合并进了 llm-d-router，由它把一个请求依次推进 encode、prefill、decode 三个阶段——多阶段流水线变成一条流，而不是留给调用方自己缝。

#!/usr/bin/env bash
# Watch a rollout from the outside. Run this in one terminal, roll the
# DisaggregatedSet in another, and read the error count at the end.
set -uo pipefail

URL="${1:?usage: rollout_watch.sh http://llm-d-router.default.svc/healthz}"
DURATION="${2:-600}"
INTERVAL=0.5

ok=0; bad=0; slow=0
end=$(( $(date +%s) + DURATION ))

while [ "$(date +%s)" -lt "$end" ]; do
  start=$(python3 -c 'import time;print(time.time())')
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$URL" || echo 000)
  elapsed=$(python3 -c "import time;print(round(time.time()-$start,2))")

  if [ "$code" = "200" ]; then
    ok=$((ok+1))
    # Slow but successful is still a client-visible outage during a rollout.
    awk -v e="$elapsed" 'BEGIN{exit !(e > 2.0)}' && slow=$((slow+1))
  else
    bad=$((bad+1))
    echo "$(date +%H:%M:%S) $code ${elapsed}s"
  fi
  sleep "$INTERVAL"
done

echo "ok=$ok errors=$bad slow_over_2s=$slow"
[ "$bad" -eq 0 ] || exit 1

# Any non-zero error count here means the revision-aware gating is not doing
# its job: a request reached a prefill/decode pair before both sides were ready.

四、Agent 感知：缓存本身就是负载

Agent 流量的形状很特别：大部分算力花在处理上下文上，而不是产出最后那几个 token，而且同一份上下文会一轮一轮地回来。1.0 原生支持这种流量。会话感知路由把多轮对话钉在已经持有它 KV 缓存的那个副本上——session-state-producer 插件给请求打上会话状态，路由器把后续轮次送回同一个后端。这种亲和性在饱和前一直保持，饱和后再均衡：拿一点缓存局部性换余量，而不是丢请求。轮次优先级调度让路由器能判断一个请求处在对话的第几步，于是活跃的 Agent 回路保持响应，而不是把每一轮当成孤立调用。原生的 Responses API 状态，包括 previous_response_id，让服务端跨轮携带上下文，Agent 不必每轮重发整段历史。对做 Agent 的人来说，实际效果是延迟更低、成本更低，而应用代码不用改。

# Does session affinity actually pay? Ask the server twice with the same
# conversation and once with a cold one, and compare second-turn latency.
import time, uuid
import httpx

BASE = "http://llm-d-router.default.svc/v1"
LONG_CONTEXT = "Line item %d: quantity, unit price, tax code.\n"
CONTEXT = "".join(LONG_CONTEXT % i for i in range(400))

client = httpx.Client(base_url=BASE, timeout=120)


def turn(payload):
    t0 = time.perf_counter()
    r = client.post("/responses", json=payload)
    r.raise_for_status()
    return time.perf_counter() - t0


def warm_conversation():
    first = client.post("/responses", json={
        "model": "open-model",
        "input": "Here is a price list:\n" + CONTEXT + "\nReply with OK.",
    }).json()
    # The id is what lets the server carry context instead of you resending it.
    return first.get("id")


# Warm session: second turn reuses the KV cache pinned to that replica.
sid = warm_conversation()
warm = turn({"model": "open-model", "previous_response_id": sid,
             "input": "Now give me the total number of line items."})

# Cold session: same question, no history to lean on.
cold = turn({"model": "open-model",
             "input": "Here is a price list:\n" + CONTEXT +
                      "\nGive me the total number of line items."})

print(f"warm second turn: {warm:.2f}s")
print(f"cold equivalent:  {cold:.2f}s")
print(f"ratio: {cold / warm:.1f}x")

# On agent traffic this ratio is the whole argument for session-aware routing.
# If it lands near 1.0, either the plugin is not tagging requests or the
# conversation is too short for the cache to matter.

五、会被反复引用的数字，以及它们从哪来

性能这件事在 1.0 之前就已经证明了。Google Cloud 把 GKE Inference Gateway 建在 llm-d 的 Endpoint Picker 上，报告 Vertex AI 生产环境里首 token 延迟下降 35% 以上，前缀缓存命中率从 35% 翻到 70%。AWS 在 SageMaker HyperPod 上跑 llm-d 的分离推理，报告相比标准 vLLM 每秒 token 数最多提升 70%。Tesla 与 Red Hat 在 AMD MI300X 上跑 Llama 3.1 70B，报告输出吞吐 3 倍、首 token 时间减半。IBM Research 用自建的开源前沿模型服务编码 Agent，报告 544 张 H100 上最多 3000 个并发 Agent、前缀缓存命中率约 98%、约 0.11 美元/百万 token，他们认为这比专有栈便宜约 11.5 倍。DigitalOcean 在 AMD 与 NVIDIA 混合硬件上跑万亿参数模型，JetBrains 的 Junie 跑在它后面，Mistral 用它做区域推理。以上每一个都是厂商自己发布的数字——所以在引用之前，正确做法是在你自己的流量上复现出同样的形状。

# Self-host or call an API? The comparison turns on prefix reuse, so model it
# instead of arguing about it. All inputs are yours; only the arithmetic is fixed.
from dataclasses import dataclass


@dataclass
class Scenario:
    name: str
    requests_per_day: int
    prompt_tokens: int          # average prompt size
    output_tokens: int          # average completion size
    prefix_hit_rate: float      # measured, not assumed
    gpu_hour_usd: float         # your actual rate, including idle headroom
    gpus: int
    tokens_per_sec: float       # measured at your concurrency
    api_in_usd: float           # per million input tokens
    api_out_usd: float          # per million output tokens


def self_host(sc: Scenario, days: int = 30) -> float:
    return days * 24 * sc.gpus * sc.gpu_hour_usd


def api(sc: Scenario, days: int = 30) -> float:
    reqs = sc.requests_per_day * days
    # Cached prefix tokens are usually billed at a discount; the point of the
    # model is to make that assumption explicit, not to bury it.
    billed_in = reqs * sc.prompt_tokens * (1 - sc.prefix_hit_rate * 0.9)
    billed_out = reqs * sc.output_tokens
    return (billed_in / 1e6) * sc.api_in_usd + (billed_out / 1e6) * sc.api_out_usd


def utilisation(sc: Scenario, days: int = 30) -> float:
    reqs = sc.requests_per_day * days
    work = reqs * (sc.prompt_tokens + sc.output_tokens)
    capacity = sc.gpus * sc.tokens_per_sec * days * 24 * 3600
    return work / capacity


if __name__ == "__main__":
    sc = Scenario(
        name="coding-agents", requests_per_day=40_000, prompt_tokens=18_000,
        output_tokens=900, prefix_hit_rate=0.6, gpu_hour_usd=2.5, gpus=8,
        tokens_per_sec=2_500, api_in_usd=1.0, api_out_usd=4.0,
    )
    print(f"{sc.name}: self-host ${self_host(sc):,.0f}/mo  api ${api(sc):,.0f}/mo "
          f"util {utilisation(sc):.0%}")

    # Read the utilisation column first. Below roughly a third, the GPUs you are
    # paying for are mostly idle, and no routing improvement fixes that.
    # Above it, prefix reuse at 60% plus flat per-token pricing is what makes
    # the self-hosted number stop being a rounding error.

六、采用之前，先想清楚你要接手运营什么

一个给你租户、优先级和零停机升级的控制面，同时也把「运营它」这份工作给了你。默认配置刻意保持向后兼容，也就是说一份没人动过的安装不会保护你的付费档——策略得你自己写，并且要在饱和状态下测过；而单租户的压测对公平性什么都证明不了。迁移之前有两件事值得先量：真实的 prefix 复用率，因为这里所有成本论证都建立在它上面；以及多轮流量占多大比例，因为会话亲和只有在对话真的会回来时才划算。如果这两个数字都不高，托管端点仍然是更便宜的答案，选它不丢人。