vLLM and SGLang Now Run Natively on TPU Through PyTorch. The Number to Remember Is 2.01 Cents

On October 20, 2026, Google shipped native PyTorch backends for vLLM, built with Inferact, and for SGLang, built with RadixArk, both on top of TorchTPU. The headline is not that these engines run on TPU, because they already did through a JAX-based backend. The headline is that developers asked to customize and debug their serving stack in PyTorch, and now the worker model, scheduler, KV cache manager and API server of each framework stay intact while TPU implementations live underneath. The published benchmark claim is a cost per million tokens of 2.01 cents for Qwen 3.5 397B-A17B on Ironwood TPU. Whether that number applies to your workload is a separate question, and it is answerable before you move anything.

vLLM and SGLang Now Run Natively on TPU Through PyTorch

1. What TorchTPU Actually Is, Without the Marketing

TorchTPU makes the TPU a native PyTorch device. Tensors are standard PyTorch tensors on a tpu device with no wrapper types, code runs eagerly or compiles through torch.compile using XLA, and collectives go through torch.distributed. Because each device runs its own PyTorch process, both engines keep their own framework machinery and add TPU implementations underneath. Most models use the standard upstream PyTorch definitions, with TPU-specific work concentrated in kernels and layer implementations, and custom kernels written in Pallas are called directly from PyTorch as custom operators inside the compiled graph. That last design decision is the one to care about: it means TPU optimization work does not fork your model code, so an upstream model update does not turn into a merge conflict.

# Get a server up from the official repos, then read their README for
# the current flags -- the engines are moving weekly.
set -euo pipefail
git clone https://github.com/vllm-project/vllm-torchtpu.git
cd vllm-torchtpu
python -m venv .venv && . .venv/bin/activate
pip install -U pip
pip install -e .

# Serve an OpenAI-compatible endpoint. Model choice must be one of the
# supported families (Qwen, DeepSeek, GLM, Kimi, GPT-OSS, Gemma).
export VLLM_TARGET_DEVICE=tpu
vllm serve Qwen/Qwen3.5-8B-Instruct --port 8000 --max-model-len 32768

# Enable host-side KV offload if you expect long prefixes that outlive HBM.
# Flag names live in the repo's docs -- check them, do not copy from blogs.

2. The Features That Matter for Agent Workloads

Google's own explanation of why this launch matters is the right one: agents repeatedly resend expanding multi-turn conversations, so serving them efficiently means finding the cached prefix instead of recomputing it. Both engines keep reusable prefixes in HBM and, with host offload enabled, move older prefixes to host memory as a second tier. The supported list is unusually complete for a first release: automatic prefix caching including hybrid models, KV cache offload to host memory and to Google Cloud Managed Lustre through TPU Sync, tool calling and structured output, speculative decoding with MTP, EAGLE-3, DFlash, DSpark and n-gram drafters, data-parallel attention with expert parallelism, context parallelism for prefill and decode, FP8 KV cache, disaggregated prefill and decode across hosts, vision-language and embedding models, and FP8, FP4, INT8 or INT4 checkpoints.

# Does prefix caching actually pay on YOUR traffic? Measure cold vs warm TTFT.
# OpenAI-compatible endpoint, streaming on, so the first chunk is the signal.
import json, time, urllib.request

ENDPOINT = "http://127.0.0.1:8000/v1/chat/completions"

def ttft(messages, model="Qwen/Qwen3.5-8B-Instruct"):
    body = json.dumps({"model": model, "messages": messages,
                       "stream": True, "max_tokens": 64}).encode()
    req = urllib.request.Request(ENDPOINT, data=body,
                                 headers={"Content-Type": "application/json"})
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=180) as r:
        for line in r:
            if line.startswith(b"data: ") and b"[DONE]" not in line:
                return time.perf_counter() - t0
    return float("nan")

prefix = [{"role": "system", "content": "You are a build log analyst. " * 300}]
cold = [prefix[0], {"role": "user", "content": "Summarize run %d in 3 lines." % i}
        for i in range(1)]
warm = [prefix[0], {"role": "user", "content": "Summarize run %d in 3 lines." % i}

c = ttft(cold)
w = ttft(warm)                     # identical prefix, second call
print(f"cold_ttft={c*1000:.0f}ms warm_ttft={w*1000:.0f}ms speedup={c/w:.2f}x")

# If warm is not meaningfully faster, prefix caching is off, or your traffic
# never repeats a prefix. Both are fixable; the number you cannot fix is none.

3. TPU Sync Is the Part Nobody Will Write a Blog Post About

Two of those features move the KV cache off the device that computed it. Disaggregated serving puts prefill on one host and decode on another, so each request's KV cache has to travel between them. Tiered offload pushes the part that no longer fits in HBM down to host DRAM and secondary storage, then pulls it back when a later request reuses the prefix. Either path adds directly to latency unless the transfer overlaps computation. Both engines use TPU Sync for this, an open-source C++ library that reads the device buffers behind PyTorch tensors directly, so no staging copy is needed on the host, and runs transfers on its own threads outside Python while prefill and decode continue. In vLLM it plugs in through the KV connector interface; in SGLang it serves as the disaggregation transfer backend and the I/O backend for HiCache.

# Turn your own traffic into dollars per million tokens, not vibes.
# Inputs: measured prompt/completion tokens per request and your hardware cost.
python3 - <<'PY'
import json

HW_COST_PER_HOUR = 4.20          # your actual rate, from the invoice
REQUESTS = [
    # prompt_tokens, completion_tokens, seconds
    (12500, 480, 2.9),
    (12510, 460, 1.1),           # warm prefix, same conversation expanded
    (800,   300, 0.7),
]

total_s = sum(r[2] for r in REQUESTS)
p_tok = sum(r[0] for r in REQUESTS)
c_tok = sum(r[1] for r in REQUESTS)

cost = total_s / 3600 * HW_COST_PER_HOUR
per_m = cost / ((p_tok + c_tok) / 1_000_000)

print(f"requests={len(REQUESTS)} prompt={p_tok} completion={c_tok}")
print(f"wall_seconds={total_s:.1f} cost_usd={cost:.4f} per_million_tokens=${per_m:.4f}")

# Compare like with like: same model, same sequence shape, same batch policy.
# A cents-per-token figure you cannot reproduce is a marketing number.
PY

4. The Numbers, and What They Do Not Tell You

The benchmarks are public through the SemiAnalysis InferenceX project, which is the part that makes them checkable. For Qwen 3.5 397B-A17B at an 8k input and 1k output sequence in FP8, Google reports the lowest cost per million tokens overall at 2.01 cents, with Ironwood TPU leading across most of the Pareto frontier, plus new state-of-the-art results on AgentX, the project's track for agentic long-context and multi-turn workloads. Separately, a Pallas megakernel for the 2.8-trillion-parameter Kimi K3 runs the entire decode step as one kernel and serves more than 700 tokens per second to a single user on Ironwood. The caveat is the one every inference benchmark has: those are specific models on specific sequence shapes. Your cost per token depends on your prompt length distribution, your cache hit rate, and whether you can actually keep a prefix warm.

# Disaggregated serving only pays if the KV transfer overlaps compute.
# Sanity check that prefill and decode hosts are actually talking to each
# other, and that transfer time is not serialized into your TTFT.
ssh prefill-host 'ss -tnp | grep -E ":7100|tpu-sync" | head'
ssh decode-host  'ss -tnp | grep -E ":7100|tpu-sync" | head'

# Then measure the boundary directly: short prompt (transfer-light) versus
# long prompt sharing a cached prefix (transfer-heavy). If the long-prompt
# case degrades superlinearly, the transfer is on the critical path.
python3 traffic_shape.py --prompt-mix short,long-cached,long-cold --n 60

# The thing to watch is not average latency. It is p99 for long-cached
# prompts, because that is the shape agents actually generate.

5. Test It as a Cost Question, Not a Framework Question

The mistake to avoid is benchmarking tokens per second on a fresh single-turn prompt and calling it a decision. Agents resend context; that is the workload you have. So measure two things: time to first token with a cold prefix, and time to first token with a warm prefix of the same length, then compute the effective cache hit rate your traffic actually produces. If the hit rate is low, prefix caching becomes a cost you pay in memory for no benefit, and the 2.01 cents figure is not your figure. The scripts below are deliberately framework-agnostic: one launches a server from the official repos, one measures cold versus warm TTFT over the OpenAI-compatible endpoint, and one turns your own token counts into a cost per million so you can compare against whatever you run today.

# Before moving a workload, prove the hit rate is real over a day of traffic.
# Parse your gateway logs; every field here should already exist.
python3 - <<'PY'
import json, collections

hits = collections.Counter()
prompt_tokens = 0
records = 0

with open("/var/log/llm-gateway.jsonl") as fh:
    for line in fh:
        try:
            r = json.loads(line)
        except json.JSONDecodeError:
            continue
        records += 1
        prompt_tokens += r.get("usage", {}).get("prompt_tokens", 0)
        # Most gateways expose cached tokens; name varies by vendor.
        cached = (r.get("usage", {}).get("cached_tokens")
                  or r.get("usage", {}).get("prompt_tokens_details", {}).get("cached_tokens")
                  or 0)
        hits["cached" if cached else "cold"] += 1

total = sum(hits.values()) or 1
print(f"requests={records} prompt_tokens={prompt_tokens}")
print(f"cached_requests={hits['cached']} share={hits['cached']/total:.1%}")

# Rule of thumb: below roughly a third, prefix caching is a memory cost
# with no throughput payoff, and you should fix the traffic shape first.
PY

6. What This Does Not Replace

The JAX-based runtimes, vLLM-Jax and SGLang-JAX, remain maintained and supported, so nobody's existing deployment is being deprecated out from under them. Model coverage today spans Qwen, DeepSeek, GLM, Kimi, GPT-OSS and Gemma 4, with day-0 support for new open models as the stated ambition rather than a guarantee. And the harder constraint is not software: if your workload lives on another accelerator vendor, a cheaper cents-per-token figure on hardware you do not own is trivia. The honest reading is that TPU inference just became plausible for teams that refused it for tooling reasons, which is most teams that refused it. Try the cost model on your own traffic before you read another benchmark.