Evergreen Tools
|
← Back to AI Tools

TorchTPU Is Open Source: Native PyTorch on TPUs

Google open-sourced TorchTPU at PyTorch Conference 2026, making TPUs a native PyTorch target alongside GPUs and CPUs. It was co-developed with the PyTorch team at Meta, so existing workloads run with minimal code changes; debug in eager mode, then move to torch.compile for production, with vLLM, SGLang, llm-d, TorchTitan, Ray, TorchRec, Hugging Face and LiteRT all wired in

Tool Interface

Interactive tool will be available soon

Features

  • ✓ TPUs become PyTorch's third native target, so one codebase covers training, serving and deployment instead of locking you to a single vendor's accelerators
  • ✓ Co-developed with Meta's PyTorch team: run existing PyTorch workloads through familiar APIs with minimal code changes, debug in eager mode and move to torch.compile for production
  • ✓ Serving side connects to vLLM (continuous batching, quantization, multi-chip parallelism) and SGLang (RadixAttention KV cache reuse), keeping upstream workers, schedulers and API servers intact
  • ✓ In Hugging Face Transformers v5, deploying to TPU is a single device_map=tpu; Diffusers runs natively with eager mode, torch.compile and tensor parallelism
  • ✓ TorchTitan reached up to about 52% MFU on Qwen3 8B, with near-linear scaling verified across up to 8,192 Ironwood TPUs (TPU v7x)
  • ✓ LiteRT joins the PyTorch ecosystem: one litert-torch.convert() call deploys to Android, iOS, macOS, Windows, Linux, web and IoT devices

How to Use

  1. Clone google-pytorch/torch_tpu on GitHub and follow the setup docs at cloud.google.com/tpu/docs/pytorch-on-tpu
  2. Get your existing model running in eager mode first, confirming operators, distributed primitives and diagnostics behave the same as they do locally
  3. Once that passes, switch to torch.compile to raise TPU utilization and cut memory; for Hugging Face models just pass device_map=tpu
  4. Pick a serving backend for production: vLLM for throughput, SGLang for structured and multi-step agentic work, and llm-d 1.0 prefix-cache-aware routing when spanning TPUs and GPUs

FAQ

What is TorchTPU?

It is Google's native PyTorch stack for TPUs, announced as open source at PyTorch Conference 2026 on 21 October 2026, with code at github.com/google-pytorch/torch_tpu. Google frames it as making TPUs a native option for PyTorch alongside GPUs and CPUs. Source: https://cloud.google.com/blog/topics/ai-infrastructure/torchtpu-is-now-open-source-run-native-pytorch-on-tpus

How much of my PyTorch code has to change?

Per Google, existing PyTorch workloads run on TPUs using familiar APIs and minimal code changes. In practice you debug in eager mode, then use torch.compile to optimize TPU utilization and reduce memory overhead.

Which frameworks and libraries are supported?

Google lists vLLM, SGLang, llm-d, TorchTitan, Ray, TorchRec, Hugging Face and LiteRT. vLLM support was built with Inferact and SGLang with RadixArk; both preserve the upstream worker models, schedulers and API servers.

How large can training scale?

Google reports up to about 52% MFU on Qwen3 8B through TorchTitan's TorchTPU path, with near-linear scaling and stability across up to 8,192 Ironwood TPUs (TPU v7x). TorchTitan ships recipes for Llama 3 70B/405B, Qwen3 8B/32B, Qwen3 235B MoE and FLUX.1 12B.

What about on-device deployment?

LiteRT (formerly TensorFlow Lite) now joins the PyTorch ecosystem. litert-torch's convert() builds on torch.export() and Core ATen operators so model code stays in PyTorch, and converted models run on Android, iOS, macOS, Windows, Linux, web and IoT, accelerated on CPUs via XNNPack and GPUs via ML Drift.