Here
is how you know that GenAI training and GenAI inference are very different
computing and networking beasts, and diverging more with each passing day:
Google has just forked its Tensor Processing Unit, or TPU, designs for these
two workloads, the very first time in more than a decade that TPU systems of
the same generation were truly architecturally distinct from each other.
To
be fair, Google has had versions of prior TPUs, specialized and homegrown AI
accelerators that the search engine, advertising, media mogul, and now AI model
maker first put into the field way back in 2015. Google got into the TPU racket
because there was no way machine learning algorithms could be added to its
applications using CPUs or GPUs without doubling its datacenter footprint. But a
lot of the time these different TPUs were just bin sorts on a designs that were
primarily aimed at AI training, with the lower bin parts being used for
inference. All of this machine learning was much simpler than the processing
used in GenAI mixture of expert models.
As
Nvidia’s $20 billion acquire of Groq last December demonstrated and as
Nvidia co-founder and chief executive officer Jensen
Huang highlighted in his keynote at GTC 2026 back in March, if you want low
latency inference response time – the kind that agentic AI demands, not the
kind of batch processing that chattybots talking to people can get by with – then
you need something that chews on tokens really well (a GPU or a TPU as we have
known it) and then something else that can spit out tokens wicked fast and then
move on to the next query.
The
first workload is called prefill, and the same kind of machinery that can do
prefill can also do GenAI training, which is sometimes called pre-training to
distinguish this model weight creation process from reinforcement learning and
other kinds of AI model tuning. The second workload is called decode, and it
means instead of chewing on tokens to either understand a query and its context
or trying to find the statistical graph of linkages between tokens in a corpus
of data, the system is spitting out tokens for the answers to a query in
whatever modality the GenAI model supports.
You
will get different opinions on just how different the architectures have to be
for the prefill and decode stages. While the new “Sunfish” TPU 8t for training
and recommendation engines (made with the help of Broadcom and shown on the
left above) and the new “Zebrafish” TPU 8i for inference and reasoning (made with
MediaTek and shown on the right) are distinct, they share many components in
their architecture. Under normal circumstances, when Google might not want to try
to do capacity planning across its datacenter fleet by splitting its distinct
hardware, it might have just added more TensorCores to the TPUs, added the
right amount of SparseCores, and added the new Collectives Acceleration Engine,
made one chip, had some of the silicon be dark depending on the workload, and
ended it there.
But
not only do GenAI training and inference (and the other workloads each
respective TPU 8 chip is handling) have different needs when it comes to
processing, they have very different needs for SRAM memory capacity and HBM
memory and memory bandwidth. (One size does not fit all, as Nvidia was well
aware, which is why we had a “Blackwell” B200 GPU for training and inference
and a B300 for inference, just to give one example.) Moreover, GenAI inference and
training have very different network needs, which is why Google has cooked up a
new datacenter fabric codenamed “Virgo” and why it is offering different scale
and topologies for training and inference workloads.
It
is not so much that the TPU 8t and TPU 8i are distinct, but that the systems
built from them are also distinct.
To
get rolling on this deep dive, here is the monster comparison table that shows
every TPU that Google has announced since 2015:
As
usual, items in black are things that we know from blogs and papers, and things
in bold red italics are things we are making informed guesses about. This table
encapsulates the data from several different blog posts as well as the
presentation by Amin Vahdat, chief technologist and senior vice president of AI
and infrastructure at Google. There are many things we do not know yet, and we
suspect many of the missing details will be revealed in a Hot Chips paper this
August, if not before then at a venue of Google’s choosing.
Pricing
for the prior “Ironwood” TPU v7e TPU instances and the not-yet-shipping TPU 8
instances have not been announced as yet, which is a shame. This is a
price/performance per watt story, and we can’t do most of the math in that
simple equation because of the missing data. But what is clear is that the cost
per unit of useful work has to come down. What Google has said publicly so far
is that the Sunfish TPU 8t has 2.7X better bang for the buck on AI training
than the Ironwood TPU v7e, the Zebrafish TPU 8i has 1.8X better performance per
watt than the Ironwood TPU v7e, and that both chips deliver 2X better
performance per watt.
We
look forward to seeing how these numbers were ginned up, and we think they are
real-world performance, not based on peak theoretical throughput as shown in
the monster above.
Let’s
Take A Look Under The Hood
Unlike
GPUs, which have dozens to tens of thousands of different kinds of cores, the
TPUs cram a lot of capability in a few cores and accelerators to get the AI processing
done. The TPU 8t and TPU 8i chips have different numbers of TensorCores, which
include both matrix and vector math units, and are different in the choice of
SparseCore or CAE accelerators that are included in the package, as well as how
much SRAM and HBM memory they have. They probably have different clock speeds
and thermals, too. But if we were Google we would push each TPU 8 chip to the
thermal limit of the TPU 8 system design, and get there by dialing up and down the
clock speed for either chip. But it may turn out that the bigger TPU 8i burns a
lot more juice and delivers the same amount of a performance boost for
inference so the increase in performance per watt for both the Sunfish 8t and
the Zebrafish 8i come to the 2X number cited above from Google.
The
TPU 8t package has two compute dies, one I/O die at the top, two other chiplets
left and right of the I/O die, and two columns of HBM3e – not HBM 4, which is
important – on each side of the compute die. The choice of HBM3e means Google
is doing a price/performance optimization on the memory. The HBM3e stacks are twelve
DRAM chips high (there is an error in the block diagram below), and the six
stacks provide 216 GB of capacity and 6,528 GB/sec of bandwidth. If you compare
to the Ironwood TPU, the Sunfish 8t chip has 12.5 percent more memory but it
delivers 11.5 percent lower bandwidth, which means it is running slower to
improve yield and lower cost.
Here
is the block diagram for the Sunfish TPU 8t chiplet:
The
Sunfish package has 128 MB of SRAM memory, meaning 64 MB on each core chiplet
in the package. This is the same aggregate SRAM per package as the Ironwood TPU
v7e. We do not know the aggregate bandwidth of this SRAM.
The
TPU 8t TensorCore has a big wonking vector unit and two matrix math units, but
there are some microarchitecture changes that are not obvious in the block
diagram above.
“TPU
8t is a powerhouse optimized for training,” explained Vahdat in his keynote. “We
have redefined performance capability by moving block scale multiplication
directly inside the MX units. This native quantization eliminates VPU overhead,
delivering nearly three times the compute performance per pod over previous
generations. This allows us to push the absolute limits of model flops
utilization at a massive scale, and reduces the training time in frontier
models. It leverages a breakthrough inter-chip interconnect technology, which
now delivers twice the bandwidth compared to Ironwood, scaling up to 9,600 TPUs
connected with a 3D torus topology.”
Let’s
not get too far ahead of ourselves with the system architecture. Back inside
the Sunfish 8t socket, there is one TensorCore per chiplet and two SparseCores
per chip, for a total of four per socket – just like Ironwood. We think Google
might have jacked up the clock speed to the neighborhood of 2.2 GHz with the
Sunfish 8t chip, which is a 42.6 percent increase over what we guessed was the
Ironwood clock speed. That may seem like a lot, but with a multi-chiplet design,
Google can increase yield on the 2 nanometer process we think it is using for
the two TPU 8 chips and use a lot of the die shrink to crank the clocks and ratchet
up raw performance and through other tweaks, get a lot more of an effective performance
boost.
The
SparseCore chips were initially designed to accelerate recommendation models
that make use of embeddings to make recommendations across classes of users.
The third generation SparseCores in the Ironwood TPU had various algorithms
encoded in them to accelerate financial and scientific calculations, and I do
not know if there are more functions in the TPU 8t SparseCores but suspect this
is so.
We
do not know if the TPU chips include the Dataflow SparseCores that were added
to the TPU v4 through the TPU v7e AI accelerators.
The
Sunfish TPU supports BF16, INT8, and FP8 formats and adds a new FP4 format that
boosts the performance by 2X over FP8 for those portions of the GenAI training
stack that can take a precision cut. The peak FP4 throughput of the Sunfish TPU
8t is 12.6 petaflops, and I presume you cut that in half for peak FP8 oomph and
then cut it in half again for peak BF16 throughput. I do not assume that INT8
performance is equal to FP8 performance, but it probably is for both TPU 8
chips.
The
Sunfish TPU 8t uses the same 3D torus interconnect as the Ironwood TPU v7e, but
it has been stretched to 9,600 TPUs in a single memory domain pod compared to
the 9,216 for the Ironwood pod.
That
brings us to the inference side of the TPU house and the Zebrafish TPU 8i.
The
Zebrafish 8i chip is a biggie, and a lot of that is because it has 384 MB of
SRAM cache across the single compute die in the 8i package. Once you have to
make a big SRAM, you might as well plunk twice the number of TensorCores on the
package and balance it out. And that is precisely what Google did. The
effective cash per TensorCore has risen by 3X compared to the Sunfish 8t and
the Ironwood v7e accelerator.
Here
is the block diagram of the Zebrafish TPU 8i chip:
The
Zebrafish 8i chip has two TensorCores but no SparseCores. It does have the new
Collectives Acceleration Engine, or CAE, which is kind of collectives offload
engine. Here is how Google described it in the technical blog:
“To
solve the sampling bottleneck, TPU 8i uses the CAE, which aggregates results
across cores with near-zero latency, specifically accelerating the reduction
and synchronization steps required during auto-regressive decoding and
“chain-of-thought” processing. For each TPU 8i chip, there are two
Tensor Cores (TC) on-core dies and one CAE on the chiplet die, replacing four
SparseCores (SCs) on core dies in previous-generation Ironwood TPU. By
integrating a specialized CAE, TPU 8i further reduces the on-chip latency of
collectives by 5X. Lower latency per collective operation means less time spent
waiting, directly contributing to higher throughput required to run millions of
agents concurrently.”
I
look forward to finding out more about how this CAE thingamabob works.
Given
the Zebrafish 8i chip’s focus on inference, Google came up with a new hierarchical
network topology for the interchip interconnect (ICI) that it has used to link
machines at the board level and sometimes at the rack level in its TPU systems.
Both
of these TPU system boards, which have four sockets per board, use Google’s own
Axion Arm server CPUs as hosts. Google has not been precise about which of the two
flavors of Axion it is using – the one based on Neoverse V2 cores or Neoverse
N3 cores. We will try to find out.
With that, in the second part of this story, I will
talk about the networking enhancements in the TPU 8 systems. Stay tuned.
