Compute Clusters Break Out Of National Labs To Scale AI

For compute-intensive applications, clusters of nodes that act as a single computer offer a way to scale performance and provide a workaround for organizations that lack access to leading-edge AI chips. Key Takeaways: Experts At The Table: Semiconductor Engineering sat down to discuss compute cluster definitions, design challenges, and use cases with Satadal Bhattacharjee, global head of cloud and AI infrastructure silicon at Arm; Ashish Darbari, CEO of Axiomise; Moshiko Emmer, distinguished engineer at Cadence; Sharad Chole, chief scientist at Expedera; Cameron Brunner, head of HPC development at Siemens EDA; and Sumit Vishwakarma, director of strategic marketing at Synopsys. What follows are excerpts of that discussion. L-R: Arm’s Bhattacharjee; Axiomise’s Darbari; Cadence’s Emmer; Expedera’s Chole; Siemens EDA’s Brunner; Synopsys’ Vishwakarma. SE: How do you define a compute cluster, and what does a chip architect need to know? Bhattacharjee: When we think about a compute cluster, it’s usually a group of interconnected servers across multiple racks, typically connected through Ethernet right now, though we are slowly moving to optical as well. The idea is that the workload that it’s serving is so large and it needs so much compute horsepower that single servers, or a rack of servers, cannot do the job. That’s when the need arises to stitch multiple servers together. This is also referred to as a pod, where you have multiple of these clusters together. That’s what a compute cluster typically is, and the idea is to have low latency so that network latency does not spike as the workload spans from one rack to the other. That’s very important. When traffic flows from one server to another, and from one rack to another, many optimizations are required to maintain performance efficiency without sacrificing it as this traffic flows across the cluster. From a chip architect’s standpoint, they have to think about cluster behavior, not just a single server, so that the efficiency that comes with faster components doesn’t get lost due to networking bottlenecks, for example, in rack-to-rack communication. The architect needs to think about network congestion, power delivery, thermal limits, software scheduling, and stuff like that. Brunner: From my background of managing software, such as multiple workload managers that manage high-performance compute clusters and executing EDA workloads at massive scale, the compute cluster is a bunch of machines that get put together and linked together in a way to solve a need at a given site operator. That could be a chip designer who’s looking to really be able to design and iterate fast with a large number of developers. A lot of work needs a large amount of capability. There are also other industries, such as life sciences, manufacturing, and CAE (computer-aided engineering), where they run very large parallel simulations. Having an infrastructure that can facilitate that is important. To Satadal’s point around high-speed interconnects, that’s very, very important in large-scale MPI (message passing interface) jobs. A cluster of machines can be much larger than you could ever have in a given machine, able to run very large workloads, or a very large number of small workloads to meet the needs of the end users. That’s something we see a lot in EDA. Often there will be hundreds of thousands, or millions of tasks that are used to build the flow when designing a chip. These may only run for a few seconds, but how do you run hundreds of thousands? Then you have multiple engineers running millions of jobs a day. It becomes more capacity than a single machine can handle, more data than you could ever put on a single disk. It’s about managing the networking, the storage, the overall compute capacity, and providing this parallel massive resource to drive your business forward. And a very important part is having the software on there to make sure you can map the needs of your business and of your activity to efficiently utilize the hardware and get the best return out of the infrastructure. The compute cluster is that whole ecosystem — the hardware, the software, all built together to solve the needs of the operator. Vishwakarma: The world is changing the cluster workloads, such as EDA workloads. How our brain evolved is similar to how the AI brain is evolving. In the past, humans did mediocre jobs, but as our brains evolved, we began doing more complex tasks — and we still only use 20 watts. Look at today compared to where AI was in 2018. The types of tasks it can do have evolved so much. From a single text prompt, you can create a one-minute video, or some people are even making movies using AI, just by using prompts. Think about how complex compute is happening behind the scenes. The LLMs have to create the next pixel to generate pixel-to-pixel, pixel-to-frame, and frames connect to become a movie, so can one GPU do it? No. Think about workloads like agentic flows. Everybody’s talking about agentic AI, where the task is not just one task. It’s a task where one agent has to compute something and give the output to the other agent, and another agent. Let’s say you have to create a research report. Right now, you give the task to the agent. ‘Hey, give me a research report, create a presentation.’ It’s going to go to a website, look for the data, crunch the data, try to find out what is best, put it into Excel, create a graph, bring that graph into PowerPoint, make it visually appealing — you can see all the agentic flows. Because of all those, we need compute beyond the GPUs, beyond the server, beyond one node into multiple nodes, which are pods. So it becomes a whole set of compute nodes, which is one brain just doing one big task. Chole: My take on this is more application-centric. We are basically looking at it as if there is a compute platform available — all the hardware, connectivity, and software stack. From this platform, we can determine how applications are launched and control access in terms of network connectivity, whitelisting/blacklisting, and access ports. And all of that is given as a service. When I say services, it is not just a cloud provider. Even internally, when we need to access a very large group of machines or hardware, it has to be accessed as some sort of service, even for internal employees. That’s basically how applications can be easily deployed, and easily managed or orchestrated. There is a failover that needs to be addressed, and it’s enabling applications to be scaled, deployed seamlessly, and secured within the given framework. This is like a boundary on top of which you can say that there is an application-level cluster, which means that you’re serving a single neural network, a single LLM, for example. You might have disaggregated prefill and decode, and you’re deploying that on a set of GPUs, maybe a two-server node, or a five-server node, and that can become a cluster by itself. That can be defined as a cluster. Emmer: From a chip architect’s point of view, a compute cluster is not just a collection of servers. It’s a hierarchy of communication domains — die-to-die, package-to-package, accelerator-to-CPU, node-to-node, and rack-to-rack. If you don’t design the lower layers with the cluster in mind, the software ends up paying for that in latency, synchronization overhead, and power. For me, cluster architecture starts at the package boundary, not just at the network boundary. The package should be treated as the first network hop. The architectural question is not only ‘how fast is the compute?’ but ‘how many boundaries does the data cross before useful work happens?’ Chiplets matter because they let you optimize local bandwidth density while still scaling externally. Darbari: From my perspective, a compute cluster is a group of machines connected and managed so that, from the workload’s perspective, they operate as one much larger computing system. Each node has its own processor, memory, and software stack, while a high-speed network and cluster software coordinate them into a single logical resource. Correctness therefore depends not only on local microarchitecture, but also on the guarantees the broader system makes about communication, ordering, data visibility, and recovery from faults. As formal verification experts, we would view this verification as modeling and capturing it as explicit contracts rather than leaving it implicit among hardware, firmware, and system software. SE: Alibaba recently announced a 10,000-card compute cluster for a data center to serve the government, health care, and advanced manufacturing industries. While China-made AI chips currently trail behind Nvidia’s most advanced chips, large-scale computing clusters with innovative and efficient network architecture can improve computing performance, enabling China to catch up. Is this a solution for U.S. companies too? What are the tradeoffs? Chole: When you’re talking about a 10,000-node cluster that Alibaba is launching, that becomes a very big challenge. Other than training jobs, there is no single job that you can run on that big of a cluster — currently at least. Maybe there are big companies with that much inference demand, but training typically requires large clusters to be available and pipelined. When you’re thinking about this problem, we think about parallelization — how the work can be broken down in terms of different parallelization strategies, and what kind of plumbing needs to be done. This is plumbing for data coming in, communication between the nodes, how the output needs to be rendered, what state needs to be saved, and what are the storage accesses? The entire connectivity between so many nodes becomes a critical point. Darbari: Alibaba’s 10,000-card cluster is a system-level answer to a device-level gap. If individual chips are not leading-edge, performance can still be recovered by scaling out aggressively and by building a fabric and software stack that makes many devices operate as one large AI platform. That strategy is valid, but it is not unique. I believe that U.S. hyperscalers have already built systems in this class. Microsoft has publicly described an Azure supercomputer for OpenAI with around 10,000 GPUs connected as a single coordinated system. AWS offers EC2 UltraClusters designed to scale AI training to tens of thousands of GPUs, including 10K‑class deployments. Google has likewise built very large TPU pod systems with thousands of TPUs for frontier‑scale training, even though the accelerator technology differs. So the core idea is already established in the U.S. — use large, tightly connected clusters to deliver competitive effective compute. The more meaningful difference is not the headline device count but the maturity of the surrounding stack. In the Chinese case, the emphasis is often on offsetting a per-chip disadvantage through system architecture and networking. In the U.S., those large clusters tend to sit inside a more mature ecosystem of tooling, software frameworks, operations, and long-running experience with distributed infrastructure at scale. The tradeoff is therefore straightforward. Scaling out can compensate for weaker individual devices, but it raises complexity sharply in performance tuning, failure handling, software orchestration, and security assurance. If those system-level behaviors are not specified rigorously, more scale can magnify uncertainty rather than reduce it. If they are specified and validated well, scale becomes a disciplined architectural advantage rather than a fragile workaround. SE: Compute clusters are common in AI laboratories, life sciences, high-frequency trading, and other specialized applications. They are increasingly common among hyperscalers. Are we going to see them becoming more common in other sectors, and why? Brunner: We’re talking a lot about AI uses, and what’s driving that right now. Cluster services are this hybrid intersection, which we’re seeing with services being deployed on traditional HPC clusters. For example, one of our big customers is Argonne National Laboratory, and their cluster that we run on Aurora has 65,000 GPUs on it. They will use that cluster for one job, such as world-scale weather simulation and these types of things. There are use cases for massive clusters, these leadership-class facilities, and their primary purpose of building these systems is for running work that can’t be run anywhere else. That’s the motivation. Where else could you do this simulation? That’s what they want to run. So typically it’s a much different usage pattern [than in other sectors]. [National Labs] do solve these very big challenges. It is something that we are doing in the U.S., and we have several of these exascale compute clusters already operational. We have three of them. It is a common use case that’s different from traditional EDA, with scientific computing as its foundation. Bhattacharjee: I can resonate with that, because I worked on HPC systems earlier, and what I’ve seen is that most of those HPC systems used to be deployed by large national labs for weather simulation, and different kinds of computational physical dynamics or CPD workloads. There are all kinds of heavy math use cases, where you need tons of compute to crunch math. Now, in the age of AI, that has become mainstream. That’s the reason why we are seeing the data centers going through a complete change or shift, where those 30- and 50-kilowatt racks, which is what hyperscalers traditionally used to use, are growing to hundreds of kilowatts of racks. The systems are no longer capable of delivering the compute required with air-cooled systems, so we are talking about liquid cooling. That’s what we are seeing — this huge shift, and this is not just confined to national labs anymore. Now we are talking about the hyperscalers building out data centers globally, and the rest of the industry catching up, and there is a push toward sovereignty because every country wants to ensure that they have their own LLMs for government and security and all of that. They don’t want to depend on other countries’ clouds. So we are seeing this build-out with high CapEx spend on acquiring land, getting all those systems stitched together, and then the software to run on them. I feel that’s going to be across different use cases. It will touch manufacturing, drug discovery, financial risk modeling, EDA, autonomous vehicles, you name it. Every sector is going to leverage that, because the level of productivity we are seeing — we are still in the early stages — is already overwhelming. The amount of time that was required to do things is now shrinking, so you can get more done. We are on this journey, and it’s going to be maybe a decade-long or longer, where all of these systems will be built out. And then you’ll have these models, which are getting more and more intelligent, to do the work that used to take a long time. Vishwakarma: I completely agree. For the frontier model itself, the training and the new model development, fundamentally, if you look at the workloads from the past 10 years, these were high-performance computing for weather predictions, and those were used for a lot of research projects. But what has changed is that for AI compute, the workload itself is heavily dependent on this one operation, which is matrix multiplication, the MAC (multiply-accumulate) operation. That’s why [HPC] pivoted to the GPUs, and not just a single GPU, but a matrix of GPUs, eight GPUs in one server. We are expanding this for clusters, but if you look at it from the bottom up, why are these data centers consuming so much power? It’s the GPU-to-GPU communication — there’s so much data transfer happening. The GPU to the memory — there’s a lot of data transfer happening, and the reason for that is because of these training models. You have weights and everything, and then you shift the weights in there to make sure that we have the best frontier model. Everybody is fighting for that. And the same thing is kind of happening on the inferencing side. What is different now is that power consumption has become the bottleneck, along with bandwidth—the latency. These are the two important challenges. Of course, there is reliability and safety, but power consumption, if you look at the amount of power these data centers are consuming. It’s the power equivalent to the city’s consumption. These needs are bringing new challenges, which did not exist before. Emmer: I agree clusters will become more mainstream, but not as one uniform architecture. As adoption broadens, we’ll see more domain-specific cluster composition — different memory footprints, different latency tolerances, and different ratios of control-heavy versus tensor-heavy work. Architecturally, that argues for modularity. And that’s exactly where chiplet-based design becomes strategic. Darbari: As more industries run into the limits of a single large server in terms of performance, memory footprint, or service availability, scale-out architectures become the practical next step. What was once concentrated in hyperscale and high-performance computing is increasingly relevant to enterprises adopting AI, large-scale analytics, and always-on digital services. Clusters are attractive because they let organizations expand capacity incrementally, maintain service through node-level failures, avoiding a single point of failure, and often achieve a better cost-performance balance than continually moving to larger monolithic systems. Then, formal verification is needed because as clusters spread, informal descriptions of behavior become less sustainable. Multi-team and multi-vendor systems benefit from precise definitions of consistency, failover, and isolation so that implementations can be checked against a common intent. Leave a Reply