Copper’s Grip On AI Scaling Is Starting To Slip

As AI clusters push beyond rack-scale limits, optical interconnects and circuit switching are reshaping how data centers scale. Key Takeaways: Data-center network fabrics are undergoing another architectural shift as optical interconnects assume a larger role in AI-cluster connectivity, extending into scale-up domains and influencing overall topology design. Scale-up has typically been defined as a single rack with copper interconnect, with programmers using memory semantics. That definition is getting fuzzier. “The old rule that scale-up is copper and scale-out is optical is starting to blur,” said Steven Woo, fellow and distinguished inventor at Rambus. “As data rates climb inside the rack, copper is running into physical limits around loss, reach, and power, challenging the ability of copper within a rack. From a signaling standpoint, optics can offer a cleaner scaling path at high data rates but must do so with manageable power and reliability over relatively short distances.” Lower latency drives all of this because GPUs underuse their capacity when they wait for data. “There have been a lot of reports on the efficiency of these GPUs that are maybe 25% efficient because they’re just sitting there waiting for data,” said Eric Aguilar, CEO and cofounder of Omnitron. Google, meanwhile, implemented an optical circuit switching (OCS) network with a torus configuration, and others are also considering OCS networks for other network architectures. This architectural shift also requires a more precise assessment of how scaling terminology is evolving across AI data-center networks. Scaling evolves yet further With an ever-shifting hierarchy of data-center scaling, particularly for AI workloads, this has rolled out as scale-up, scale-out, and scale-across approaches. More recently, scale-in has shown up. “This is a new term that has only started in the last three months,” said Vishal Chandrasekar, head of product management at Ayar Labs. “It refers to the amount of bandwidth coming out of the GPU that stays within a single chassis.” The definitions of these, up until recently, can be summed up as follows: Aside from the addition of scale-in, scale-up is evolving. Computing clusters are growing beyond the bounds of a single rack, recruiting servers in adjacent racks. Now the interconnect distance is longer, and copper may no longer be the best medium. As a result, fiber is percolating into scale-up. “If it’s going to be within a rack, you’re probably going to use copper,” explained Chandrasekar. “If it’s adjacent racks, you’re on the margin, but let’s say you stick with copper. If you’re going into anything more than adjacent racks, you’re going to go with optics.” In terms of how long a link requires optics, “Assuming they’re using a 200-gig signaling rate, copper probably makes sense up to five or, if you really push it, seven meters,” Chandrasekar explained. “Certainly, once you hit 10 meters and beyond, it absolutely has to be optics.” Scale-up, as previously defined, had three elements: single rack, copper, and memory semantics. The first two are now being phased out, and the remaining defining factor is the use of memory semantics. “Software guys define scale-up as one operating system domain where they have one memory,” said Priyank Shukla, director of product management, interface IP at Synopsys. “So, if you’re running a process, the processor can write to one memory location, regardless of where that memory is.” Typical implementations of scale-up and scale-out minimize the number of hops. “Within the scale-up domain, every GPU is just one hop away from every other GPU,” said Chandrasekar. “If you’re talking scale-out, it’s like two hops.” The one-hop expectation for scale-up is enforced by the new UALink standard. “UALink scale-up fabrics are designed to maintain single-switch-hop accelerator-to-accelerator connectivity,” explained Shukla. “Even as a scale-up domain expands beyond a single rack, communication between accelerators is intended to traverse only a single UALink switch. As a result, connecting accelerators across racks does not inherently require a switch-to-switch topology that would introduce an additional hop. The single switch can be located in different places within the rack architecture, such as a Top of Rack (ToR) or Middle of Rack (MoR) deployment, but the key point is that accelerator traffic still traverses only one switch between endpoints.” Once scale-up domains extend beyond the rack, the same latency and hop-count constraints shift attention from interconnect media to the broader network topology that links accelerators, switches, and servers. Network architectures The way servers are interconnected in the data center makes a power and performance difference, but there’s no single correct configuration. Different workloads benefit from different arrangements. One of the most common is called a Clos or leaf-and-spine network. Given two servers, there’s a short path between them. While this architecture is pervasive, one of the biggest players is using something else for its AI network. “Google doesn’t have a leaf and spine network,” observed Chandrasekar. “It’s a torus network, which means every TPU is linked to adjacent TPUs, adjacent in three directions: up, down, and to the right, giving it a systolic character.” Fig. 1: Network architectures. The top shows leaf-and-spine, with a one-hop path from any server to any server. Below is a torus, which requires more hops between any two servers. Source: Bryon Moyer/Semiconductor Engineering A key difference between the two is the number of hops necessary to get from any server to any other. With leaf-and-spine, it’s one. With a torus, it depends on the locations. The path above shows four hops. “If you want to get from one GPU on one end to another GPU on the other end, you have lots of hops along the way,” Chandrasekar noted. A long-enough path can be implemented with optical for better power/performance. But each hop brings some inefficiency since the typical way these networks are built involves packet switching. Packets contain artifacts in their headers that must be inspected at each switch or router. Optical technology doesn’t understand packets. So, the four-hop path above would require, at each hop, that the optical signal be converted to electrical. Once in the electrical domain, the next hop can be determined, but before the signal makes that next hop, it has to be converted back to optical. These repeated conversions at each node hurt latency and consume a fair bit of energy. Is there a better way to build optical networks? Circuit switching is old school Packet switching is taken for granted today, but it’s not the original network. Previously, there were nationwide telephone systems that used circuit switches. Circuit switching means opening up an entire path from source to destination and giving exclusive use to one flow. In the old days, this flow was the conversation between the people at both ends of the line. Party lines aside, no one else could use that line while the conversation was busy. An analogy helps here. Let’s say you want to take a road trip from Los Angeles to Boston. In real life, you’d share the road with many other vehicles, and at each intersection or interchange, you and everyone else have an opportunity to change your route to get to your specific destination. Now, as anyone who has ever been held up by a traffic light knows, such travel would be so much easier if everyone else would go away and leave the road to you. If you were to reserve the entire path from LA to Boston, eliminating all stops, you could get there much more quickly and efficiently. This is what a circuit-switched network provides. It comes with a slower background network that configures the path before enabling the conversation. For the phone system, the most recent of these is called Signaling System 7. The problem is that even while you’re still in California, no one else can be on any of the other roads along the way, even right outside Boston. Only when you’d arrived would those roads be freed up for someone else’s trip, which sounds inefficient. Packets to the rescue… for some things This is where packet switching came in, which can be thought of as the stop signs, stop lights, and highway interchanges that allow multiple people to make multiple trips to completely different destinations, all while using roads that others are also using. Replace those intersections with routers and the roads with wires, and you have the essence of a packet-switched network, which has become the standard for most of networking. What’s new is the data center challenge of repeated conversions between optical and electrical domains, due to packet routing. “Data moves by picking a fat, infinite-bandwidth fiber signal, putting it into an electrical switch, converting that optical signal to an electrical signal, reading the packet, and then routing it to another GPU,” explained Aguilar. “That optical-electrical-optical loop burns an enormous amount of power, adds latency, and drives up cost in data centers. About 15 years ago, Google started looking at this, and about 10 years ago, made it happen with a MEMS [microelectromechanical systems]-based optical circuit switch.” The company uses nothing but adjustable MEMS mirrors at each router. Those mirrors reflect the laser light coming from the source fiber onto the destination fiber for that particular route. Those are set up at every node to provide a clear path from source to destination, with no conversions necessary. This approach is called optical circuit switching (OCS). Google’s implementation Google has found OCS beneficial. “We have spent the last eight years deeply integrating OCS and wave division multiplexing (WDM) into Jupiter [their network],” noted author Amin Vahdat, senior vice president and chief technologist, AI and Infrastructure at Google, in a paper from 2022. “The combination of OCS with our software-defined networking (SDN) architecture has enabled new capabilities: support for incremental network builds with heterogeneous technologies; higher performance and lower latency, cost, and power consumption; real-time application priority and communication patterns; and zero-downtime upgrades. Jupiter does all this while reducing flow completion by 10%, improving throughput by 30%, using 40% less power, incurring 30% less cost, and delivering 50× less downtime than the best-known alternatives.” The power savings reflect a number of changes. “You’re eliminating half of the transceivers, eliminating the liquid cooling that’s required, and then the ASIC/electronics that are in the switch itself, and you’re replacing that with an optical solution,” said Aguilar. In an OCS, when a large chunk of data must flow between two servers, first the background network configures the mirrors all along the route, then the flow can proceed with no conversions or routing decisions. The key here, however, is that this data all comes from the same place and goes to the same place. None of the data can branch off and travel to a different node. That makes this an optimization that’s workload-specific. If this approach were taken for short, frequent bursts of data, more time would be spent setting up the routes and tearing them down than sending data. Using the road analogy, reserving the whole road from LA to Boston might make sense if there is a convoy that is longer than the full route. In that case, no one else can use the road while it’s occupied, but no part of it is sitting idle. This is similar to long data flows that warrant OCS. “Optical circuit switching makes sense for ‘elephant flows,’ a term that I’ve been hearing others use,” said Aguilar. “It’s much better than these bursts of ‘mouse flows.’ This is more for AI and large language-model learning and inferencing.” And since OCS depends on reserving an end-to-end optical path before payload transmission begins, its effectiveness also depends on the control plane that configures those paths. A new Signaling System A background network configures the mirrors when the flow is set up. Analogous to Signaling System 7, it’s a slow Ethernet-based control network running alongside the optical network. “You have a lower-bandwidth Ethernet network to reconfigure the switches and the data route,” said Aguilar. It doesn’t take too long to reconfigure the network, but it does take time, which is another reason this makes sense only for long flows. Even so, configuring the network isn’t what limits how long a new configuration can be up and running. “It’s the transceiver lock that’s the bottleneck,” said Aguilar. It may seem more complex to replace one network with two, but the control network isn’t complex. “You still have Ethernet that connects rack to rack to route the pipes to the right spots, but it’s a simpler, more elegant solution,” he said. It also doesn’t add significantly to capital costs. “Ethernet switches are cheap. The Ethernet switch is just a simple low-bandwidth command center.” Mixing packet and circuit The torus network in Figure 1 above uses point-to-point connections, and circuit switching benefits the many-hop connections. But the leaf/spine architecture is different: it features switches and routers. Could this architecture benefit from OCS as well? In the torus example, the flows are end-to-end. This could be done in a leaf/spine setup as well, although fewer hops mean less power savings. What some companies are reviewing is whether, for instance, the top tier of a leaf/spine network could be an OCS, with the source and destination routers accepting multiple packets on the input side, bundling them together for optical transport to the destination router, where the packets are extracted and finally sent, electrically, to their final nodes. “OCS optical will terminate into a switch, and then after that, it’s electrical,” said Synopsys’ Shukla. Fig. 2: Mixing circuit and packet switching. The top portion illustrates an end-to-end circuit in a tree-style network. If the optical path were limited to the router-to-router connection, then packets could be assembled in the source router and disassembled and rerouted electrically in the destination router. Source: Bryon Moyer/Semiconductor Engineering As long as the traffic is steady enough, with multiple packets from the source side targeting nodes connected to the final router, then it could make sense to open an optical path between the two switches. As shown in Figure 2, this may look somewhat trivial, but if the path between routers is complex and goes through many nodes, such a circuit could make sense. Google’s OCS is a done deal, and in fact, they’re moving from a torus network to yet another configuration called a dragonfly. Other companies have yet to deploy such a network, whether torus or some other tree-style option, but word on the street is that OCS is being considered by different companies for different networks. “People are looking at that saying, ‘Can we use that technology in leaf and spine structures to replace the second layer with an OCS while keeping the first layer cloud packet switching?’” said Chandrasekar. The initial switchover to optical is manageable. “There isn’t a significant overhaul in a data center to make this happen,” said Aguilar. “If I can retrofit my data centers with this solution, I can reduce the power demand, or I can increase capacity in existing data centers.” Once an optical network is in place, it doesn’t need further upgrades. The fiber has no idea what generation of transceiver, memory, or processor is in use, so changing those generations doesn’t require changing the network. This idea is still being mulled over in the industry. “It’s more in the exploratory phase right now,” Chandrasekar added. “No one’s deployed it.” Optical encroaches further Both developments involve increasing optical participation in the data center. With scale-up, it’s now joining copper for the longest connections, making the complete set of scale-up connections a mix of copper and fiber. Optical standards will help. “The OCP is trying to define the standards so that everyone can play in the ecosystem,” said Aguilar. “Google has developed a lot of proprietary aspects, and it needs to be standardized for this to be adopted, or it won’t work right.” If this takes off, will it stress optical suppliers with a sudden burst of demand? Aguilar doesn’t think so, based on planned production scaling and industry mergers. “For the next five years, based on our projections, we’re okay,” he said. “Our foundry partner is already scaling as a result of the expectation of growth, while Nvidia announced a $2 billion investment in Lumentum, and another $2 billion in Coherent for OCS.” OCS, meanwhile, provides a way to make optical more efficient for long flows between two destinations, and gives data centers a new way to configure optical interconnect. “When you’re trying to do all these memory-latency-sensitive things, you want to reduce the number of hops, and being purely in the optical domain does help,” Aguilar noted. Additionally, new alliances are forming. For instance, Lightmatter recently joined Nvidia’s NVLink ecosystem. But work remains. “Many companies are currently evaluating this option, and from a technology point of view, it makes a lot of sense,” said Ewald Liess, senior vice president of sales and business development at SignatureIP. “But with production costs and reliability, there are plenty of challenges to be solved.” Related Articles Scale-Up, Scale-Out Get a New Partner For reaching farther into another data center, developers are now talking about scale-across Multiple AI Scale-Up Options Emerge As data center infrastructures adapt to evolving workloads, parts of Ethernet can be found in scale-up approaches Verifying Scale-Up And Scale-Out In Data Centers Navigating a sea of standards and options in a rack and between racks Leave a Reply