Optical circuit switches are not new and have been used in network technology research labs and in the telecom industry in production for the past two and a half decades. But the time has come for them to move into the scale up network domain in high performance computing clusters according to Salience Labs, which has built an optical circuit switch that is based entirely on silicon photonics technology and that, ironically enough, is derived from advanced research trying to create a photonics computing platform.
Two types of optical switches have emerged and are used in production today. One is based on micro-electrical-mechanical systems (MEMS) mirror technology, where arrays of tiny mechanical mirrors spin to create circuits linking strands of fiber optic cables. Google hardwired the first three generations of its TPU AI accelerator systems together directly (just like Nvidia has done with its nodescale and rackscale GPU designs thus far), but for the past four generations since 2015, Google has used its “Palomar” MEMS devices that are part of the “Apollo” OCS to be the backbone of its TPU clusters. The OCS allows for clusters to be reconfigured on the fly, and also allows up to 9,216 TPUs to be lashed together in a coherent memory cluster.
Lumentum, which has partnered with Nvidia for various optical technologies, also sells MEMS-based OCS gear and has aspirations outside of telecom and in the AI datacenter. The other OCS technology in production is called liquid crystal on silicon (LCoS), and this approach is used by Coherent in its OCS devices. Not surprisingly, Nvidia has also partnered with Coherent for various optical technologies.
Salience Labs has not divulged its particular switching mechanism, but we know for sure from talking to the OCS startup that it is using neither MEMS or LCoS techniques to flip through circuits. The startup spun out of Oxford University in the United Kingdom and the University of Münster, and specifically is founded on the work of teams led by Harish Bhaskaran, a professor of applied nanomaterials at Oxford, and Wolfram Pernice, a professor of experimental physics at Münster. Both are co-founders at Salience Labs and a joined research effort was looking at phase change optoelectronics.
The fact that Salience Labs has partnered with Tower Semiconductor for its PH18DA integrated III-V lasers and TPS45PH low-loss silicon nitride waveguides is another indicator that it is using some sort of phase change technology to do the switching inside of a silicon photonics framework.
Whatever it is, Vaysh Kewada, co-founder and chief executive officer at Salience Labs, is not telling The Next Platform or anyone else even as the company is rolling out a 32 port OCS and is looking at delivering OCSes with 64 and 128 ports for scale up AI networks. But it could be any number of things, including a thermo-optical Mach-Zehnder interferometer, an electro-optic element of some sort, or the expected a phase change material that is the heart of the optical switching.
What matters is that it is not mirrors or LCDs, because these are very slow, taking milliseconds to reconfigure a port to port link compared to under 300 microseconds with the on-chip waveguide switching Salience Labs is likely using.
What Kewada is emphatic about is that it is time to shake up optical circuit switching and get a true silicon photonics product that can actually do scale up memory fabrics across fleets of XPUs as well as maybe some of the aggregation layers out in the scale out networks like what Google is doing with its Apollo switches.
“We chose to develop OCS because our view is that there was quite a large number of players developing solutions on the CPO, NPO, XPO front, trying to solve the question of how you get data off of the chip and onto the optical fiber,” Kewada tells The Next Platform.
And by the way, Kewada has a bachelor’s and master’s degree in physics from Imperial College London and was also an entrepreneur in residence of the Oxford Science Enterprises incubator.
“But what was interesting to us was that this trend was coming down the line,” Kewada continues. “When there were more optical connections established in the datacenter, we thought about what the switching architecture would look like, and so we developed our products with a very simple hypothesis: There would be more optical connections in the datacenter, whether pluggable, CPO, NPO, whatever format, and when that happened, the switching layers would move towards more heterogeneous architectures, i.e. combinations of electrical packet switches in some cases and optical circuit switches in others. Our view is OCS was a market ripe for disruption, because the OCSes that are available today on the market are based off a technology that fundamentally is twenty years old if not older.”
The Salience Labs OCS is comprised of two chips. One is a fully integrated silicon photonics OCS, and the second is an amplification and signal conditioning chip. This latter chip is akin to retimers and redrivers for copper circuits where they bandwidth is high by the length of the wire keeps getting shorter because of noise issues at higher and higher bandwidths.
The two chips are mounted back to back on a PCB card, as you can see the two sides of that card in the image above. You can also see the front panel of the 32 port test system in this composite photo.
“With OCS, you want to be able to produce the product in an integrated chip format to reach volume production, giving good cost per port, good manufacturability, and all the advantages of being in an integrated format,” Kewada explains. “One of the key disadvantages of being in silicon photonics has always been the question of loss. As soon as you integrate on chip, you are going to incur some optical loss, and so we have solved that using an amplification that is our own component design and fabricating it on arrays, which let us meet the bill of materials cost goals. And that’s the crux of our solution. This amplification chip is one of a kind, and the crux of the stack. We are also doing a lot of work on our core architecture to ensure that we can continue to scale port counts.”
The OCS that Salience Labs launched in March and is now ramping in production has 32 ports, but the architecture scales to 64 ports, 128 ports, and 256 ports. The output from the OCS is 100 Gb/sec native line rate with PAM4 modulation (with two bits per signal) to get that to 200 Gb/sec effective bandwidth per lane.
What the potential customers talking to Salience Labs seem to be keen on is expanding the size of the scale up domain for an AI system, which is stuck at 72 GPUs from both Nvidia and AMD right now, and a bunch of OCS devices might help here given the low latency of the switch and the relatively low reconfiguration time. Some customers want to go beyond one rack for that coherent memory domain for GPUs and other XPUs, others want an alternative to the coherent memory fabrics currently available for rackscale machines, and still others are looking at using OCS for the first layer of networks gluing together multiple machines for the decode phase of GenAI. It would be great if these different options could be reconfigured on the fly across a row of AI systems.
The port to port hop on the initial OC-32M device that Salience Labs is now shipping is under 10 nanoseconds – that is main memory speed. The port to port hop on the Broadcom Tomahawk Ultra Ethernet ASIC is around 250 nanoseconds. That is a factor of 25X lower latency across the switch, and any system architect would want that low latency to more tightly couple memories and their associated compute. Other Ethernet switches are on the order of 450 nanoseconds to 650 nanoseconds for very good ones, and can be milliseconds for ones that are really not appropriate for scale up memory fabrics at all.
Here is how Salience Labs says it stacks up to the competition, which is Ethernet switches and MEMS-based OCS devices at least in this table:
The key thing to notice is how that reconfiguration time is much lower for the SiPho OCS switch chip than it is for a MEMS device – more than a factor of more than 3X. The energy consumption per port is 8X lower for either type of OCS compared to a relatively fast Ethernet switch – we don’t know enough about the Tomahawk Ultra yet to be more specific.
Here is a better shot at the OC-32M switch chassis:
And here is a zoom shot on the amplifier chip and the fibers coming off of it:
This particular box has one OCS socket, but Salience Labs says it can package up to eight of these modules into a single 1U chassis.
Here is the cut and dry of the argument for adoption that Salience Labs is making for the adoption of OCS in the scale up domain: You don’t have to have two different kinds of compute engines to get low latency inference. Here is how Salience Labs expects for its OCS switchery to performance along a Pareto curve for a hypothetical machine with 576 GPUs lashed together, in this case with Nvidia’s own Megatron 2T model:
That lower latency of the OCS at the scale up layer improves token per second per user throughput by 80 percent without having to move to a dual-system architecture, mostly due to the much lower latency of the memory fabric switch. By going OCS, you can scale up across multiple racks and boost throughput and still stay on GPUs.
I hope Nvidia and AMD are listening.
