Andy
Bechtolsheim, legendary co-founder of Unix system maker Sun Microsystems and
more than a few networking startups after that, has nothing against co-packaged
optics. Well, except that thus far CPO modules have not been able to be
manufactured in volume.
There
is no question that co-packaged optics is coming to datacenters at this point,
particularly with AI industry juggernaut Nvidia embracing CPO links on its
Quantum X800 InfiniBand and Spectrum X800 Ethernet switches – previewed
in June 2024, launched
in March 2025 and shipping since December 2025. Nvidia is also told customers
in March this year at GTC that with the future “Feynman” GPUs coming in 2028
that the NVSwitch 8 coherent memory scale up network at the heart of its
rackscale systems will be moving to CPO as well. This strongly implies that the
Feynman GPU and whatever the next generation of Arm server CPU that Nvidia pairs
with it – we don’t have a codename as yet – will also have CPO ports.
To
address the manufacturability issues, ahead of GTC this year, Nvidia pumped $2
billion each into Lumentum and Coherent, who among other things make the lasers
that drive CPO ports. Nvidia also inked multi-billion dollar, multi-year supply
agreements with these companies. And later in March, Nvidia
inked a similar $2 billion deal with Marvell to add NVLink Fusion ports to
custom accelerators, but we think there is a chance that some CPO
technology from the
$2.5 billion Celestial AI acquisition by Marvell back in December 2025 is
also part of this deal.
In
the meantime, AI datacenters need more density in their networks and they need optics
that are denser than the SFP, QSFP, and OSFP pluggable modules that have
dominated the datacenter for the past two decades. And more specifically, they
need more density than the OSFP pluggables that were first conceived in 2016 by
Arista Networks, the upstart who took on Cisco Systems in datacenter networking
where Bechtolsheim is co-founder and chief development officer, and Google and the
industry adopted the standard and delivered pluggables at 400 Gb/sec speeds a
year or so later. These have become the most popular pluggable optics in
history.
The
trouble is, the OSFP modules are too fat for the radix that is necessary for
modern AI systems, particularly if you want to use Ethernet for scale up and
scale out. Enter Extra-dense Pluggable Optics multi-source agreement, which is
a standard started by Arista Networks, Microsoft, Marvell, Broadcom, and Ciena that
now has over a hundred companies backing it. Google is not on the list, and
that might mean that Google is going to try to jump from OSFP pluggables to
some sort of on-chip CPO in the same timeframe that XPO modules are expected to
come to market in volume.
The
XPO module is clever in that it packs a whole lot more bandwidth into the same space
as an OSFP module, and that means the front panels on switches have a lot more radix
coming out of them as well as more bandwidth. Because this is physics we are
dealing with, you have to pay for the XPO module in heat density, and that
means the XPO module has to remove that heat with a cold plate and liquid
cooling. But this is a minor thing in a world where GPUs and XPUs need to be
connected in larger and larger scale up and scale out domains and liquid
cooling is becoming normal for rackscale systems. There is no other way to get
components closer to each other and therefore drive down latency and drive up
performance.
According
to Bechtolsheim, right now, using 1.6 Tb/sec OSFP modules, you can get 32 ports
for a total of 51.2 Tb/sec coming out of the front panel of a 1U Ethernet
switch. The OSFP modules burn somewhere between 30 watts and 40 watts, and if
you slap a cold plate on it, you can’t really increase the cooling capacity or
the switch front panel port density. What that means is if you have a 204.8 Tb/sec
switch ASIC, which believe it or not we will have soon, you need 4U of chassis
space to get 128 OSFP modules running at 1.6 Tb/sec.
The
XPO module crams 64 channels running at 200 Gb/sec into the same space taken up
by two OSFP modules, which increases lane density by a factor of four. Bandwidth
goes up by a factor of eight with the XPO module, and heat dissipation goes up
by a factor of ten to 400 watts.
Here
is the exploded view of the XPO module:
There
is some lucky geometry going on here, says Bechtolsheim, in that the paddle
cards – the little motherboards of circuits – fit into the same exact space as
two OSFP modules. So the chips and paddle card designs did not have to be
changed. You put two side by side and then stack a pair of those pairs belly to
belly and you get eight times the lanes in twice the space.
When
you look at it, XPO seems kind of intuitively obvious. Like many good engineering
ideas do in hindsight.
The
XPO module will support a variety of front panel fiber connectors:
The
upshot, says Bechtolsheim, is that XPO supports any optics standard, any optics
technology, any type of driver, retimer, or gearbox, any optical connector, and
any kind of cable, and it can give density improvements without having to shift
to CPO. Presumably the economics will be better, but don’t jump to that
conclusion too fast.
One
other side benefit is that with liquid cooling on the XPO components, they run
anywhere from 20 degrees to 25 degrees cooler (that’s in Celsius) in a 12.8 Tb/sec
ZR module, with its schematic and heat map shown below, than an air-cooled 1.6
Tb/sec OSFP-ZR module. (That is around 45 degrees Celsius for the XPO module compared
to 65 degrees to 70 degrees for the OSFP module at the same 1.6 Tb/sec
bandwidth.)
That
lower temperature means fewer failures in the field – how much remains to be
seen. But this has real monetary value in an AI supercomputer where any outage
stops a training run cold and you have to roll back to a checkpoint and start
again. Time is money when it comes to GPUs and XPUs, and a lot of money at
that.
In
the switch designs that will come out using XPO, the power for the XPO modules
will be drawn directly off of a 50 volt bus bar, rather than going through
motherboard voltage converters used in switches using OSFP modules. This is a
more efficient way to distribute the power.
The
net-net is that a rack of switches using XPU modules can cram an aggregate of
6.5 Pb/sec into an Open Rack v3 switch rack from the Open Compute Project,
which looks like this:
But
here is the money math that will have AI datacenter builders interested in the
XPO modules. They can cut the size of their datacenters in half because of the
increase in density of the network racks. The change is dramatic.
Let’s
say that you want to have a row of compute and networking based on Ethernet
that links 512 XPUs together and that you want to use switches with
top-of-the-line 1.6 Tb/sec OSFP ports. You will have 128 compute engines per
rack, and need four racks for the compute, but it will take eight racks of
networking. With a shift to XPO, you have the same four racks of XPUs, but you
only need two racks of switching. So twelve racks collapses down to six racks
for the same compute and the same interconnectivity. The switch racks are
hotter, but there are fewer of them and the power is mostly a wash. The lengths
of the cables between the XPUs and the switches are also shorter, which is a
benefit in that it is less fiber, which costs money. Over a 1 gigawatt datacenter,
such little things add up, as does having to pour less concrete and putting up
smaller shells for a given amount of compute and networking.
So
how far does XPO push out CPO? And Bechtolsheim gave us the same answer he has
given for years now.
“We
have told both customers and said in public, we are not religious about any
technology here,” Bechtolsheim tells The Next Platform. “So that is understood.
The only thing we are religious about is that we can ship stuff in high volume.
Everybody working on XPO did their own work based on their own nickel, and they
all want to own what they developed, and they all are going to be there. This
effort has been driven by this one large end customer, but I think everybody
looked at this and concluded that this the way to get the next level of density.”
There are more than 20 different vendors that are
going to be making XPO modules, and they are expected to be in volume
production in 2027.
