🔍 Read the full analysis: Getting AI Ready For TPUs With Hugging Face And TorchTPU on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face has announced TorchTPU, a PyTorch backend for running supported Transformers, Diffusers and Kernels workflows on Google TPUs. The company reports large decoding gains for two models under a specific batch-size and hardware configuration, but has not provided a full compatibility list or detailed benchmark methodology.
Hugging Face has announced TorchTPU, a backend that lets supported PyTorch workflows run on Google TPUs through its Transformers, Diffusers and Kernels libraries. The company says developers can select a TPU through familiar library interfaces, a change intended to reduce the need to adapt code for the older PyTorch/XLA route; coverage across models and operations has not yet been fully specified.
Hugging Face’s examples use existing library calls to place work on a TPU: a Transformers model can be loaded with device_map=”tpu”, while a Diffusers pipeline can be moved with pipe.to(“tpu”). The announcement says compiled execution is available as an option. It also lists Pallas kernels tuned for TPUs, distributed training support and integration with torch.compile. For large models, Hugging Face says a TensorParallelConfig can split the model across chips during loading.
The company reports decoding results for selected kernels from its Kernels Hub and Tokamax. On a TPU v6e-8 at batch size 64, it says decoding was 21.5 times faster for Qwen3-30B-A3B and 12.8 times faster for Gemma 4 26B-A4B than each model’s default implementation. These are company-reported comparisons for named models and a specific setup, not independent benchmark findings. The announcement also includes image-generation timings on a TPU v6e-4, but does not provide a complete methodology or results for other workloads.
Installation instructions are aimed at a TPU virtual machine and specify a CPU PyTorch wheel, rather than a CUDA build. Hugging Face says the TPU machine does not need the CUDA version. The announcement also says its Optimum TPU package is no longer required to run Transformers and Diffusers models using the new native support. It does not establish that every model or operation in those libraries can use TorchTPU.
A Simpler Route to TPU Experiments
TorchTPU could make it easier for teams already using PyTorch and Hugging Face libraries to evaluate Google TPU capacity. If supported models can be directed to TPU devices through familiar APIs, developers may have less code to maintain across accelerator platforms and fewer changes to make before testing a TPU deployment.
That potential is separate from whether a given workload will run well. A simpler device-selection call does not establish broad compatibility, stable operations, faster execution or lower costs. The reported speedups apply to two model configurations at one stated batch size on a particular TPU setup. They do not predict performance for different models, batch sizes, TPU generations or software stacks. For teams deciding where to run workloads, reproducible results and cost comparisons will matter alongside ease of use.
Google TPU compatible PyTorch models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
From PyTorch/XLA to TorchTPU
Before this release, Hugging Face says PyTorch users generally used PyTorch/XLA, also called torch_xla, to run workloads on TPUs. The company describes that approach as workable, while citing lazy tensor execution, graph recompilation after code changes and harder debugging as development challenges that informed its decision to introduce a native backend. These are Hugging Face’s characterizations, rather than an independent assessment of PyTorch/XLA.
Hugging Face had also offered Optimum TPU for inference and training with its models. The company says that package is no longer necessary for running Transformers and Diffusers models with the new native support. Google TPUs are specialized accelerators offered through Google Cloud, including configurations ranging from individual chips to larger pods; TorchTPU is presented as a software connection between that hardware and Hugging Face’s PyTorch tools.
“With TorchTPU, a native PyTorch backend, TPU is now just another device.”
— Hugging Face
Compatibility and Results Await Testing
The announcement does not provide a complete compatibility matrix identifying supported or unsupported models, operators and workloads. It also does not give full benchmark procedures, independent verification or a cost comparison for the reported performance figures. Results may vary with model versions, batch sizes, TPU generations and software configuration, so the cited numbers should not be treated as general performance guarantees.
Hugging Face says it is running the Transformers test suite for TPU support and working toward nightly TPU continuous integration. The announcement does not specify when that testing will be available, what coverage it will provide or whether detailed results will be published. Operational stability and debugging experience across a wider range of workloads also remain to be established.
Nightly TPU Tests Are Planned
Hugging Face says its next work includes expanding TPU testing for Transformers and adding nightly CI on TPU. Developers can consult the company’s Diffusers TPU guide for distributed and compiled usage, and follow the installation instructions for TPU virtual machines. Further information on coverage and performance will depend on the results of that work and subsequent updates from the company.
For now, developers evaluating TorchTPU will need to test their own models and configurations rather than assume that library-level support covers every workflow. A published support list, reproducible comparisons across representative workloads and clearer cost data would help teams judge whether the backend fits their deployments.
Key Questions
What is TorchTPU?
TorchTPU is a PyTorch backend that Hugging Face says enables supported PyTorch workflows to use Google TPU devices.
Which Hugging Face libraries are named?
The announcement names Transformers, Diffusers and Kernels. It does not say that every model or operation in those libraries is supported.
Does TorchTPU replace PyTorch/XLA?
Hugging Face presents TorchTPU as a native backend and says its Optimum TPU package is no longer required for running Transformers and Diffusers models with the new support. The announcement does not provide a complete account of how TorchTPU relates to every PyTorch/XLA workflow.
How broad are the reported performance gains?
The reported decoding comparisons cover two named models at batch size 64 on a TPU v6e-8, measured against each model’s default implementation. Hugging Face does not provide enough methodology or results to generalize those figures to other models or deployments.
What information is still missing?
A full compatibility list, detailed benchmark methodology, independent comparisons and cost data are not included in the announcement. Hugging Face says it is working on TPU testing and nightly CI, but gives no timeline for those efforts.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
