The 1

Is it better to cram more compute into each rack or rethink the architecture? Key Takeaways: Data centers are gearing up for a future in which a single rack could consume as much power as an entire building. Hyperscalers already are collaborating to establish standards for 1-megawatt racks, and as AI workloads grow more continuous, autonomous, and power-intensive, there is an ongoing debate about whether developing extremely dense racks is a necessary step or wasteful expense. Technologies like optics, specialized silicon, and high-voltage DC systems can enable a more fragmented future with distributed workloads and reduced density. At least for now, data center operators and cloud service providers (CSPs) are preparing for the move to 1-megawatt racks, which will require foundational changes to the data center architecture. Google, Meta, and Microsoft are collaborating within OCP to establish standards supporting the transition through a project called Mount Diablo, based on OCP’s Diablo framework. This includes various components such as cooling, rack infrastructure, and other elements. These CSPs aim to unify these elements within OCP, although each CSP will follow its own path, striving for basic collaboration and standardization around such concerns as liquid management, water flow, and power supply, with many OCP standards addressing these aspects. Today’s highest-performance AI systems are constrained by short-reach, high-lane-count copper connectivity, together with the power delivery, networking, and cooling infrastructure required to support them. “As AI compute density continues to increase, the number of electrical and signaling connections required to scale these systems grows dramatically,” said Arif Khan, vice president of product management and marketing in Cadence’s Silicon Solutions Group. “At the same time, GPUs are becoming larger and more power-hungry, driving corresponding increases in memory capacity, cooling infrastructure, and power delivery requirements. The result is a fundamental shift in datacenter architecture. The rack can no longer be viewed as a passive enclosure containing a collection of servers. Instead, it must be designed as an integrated compute system.” Khan says this philosophy is already visible in rack-scale architectures, such as Nvidia’s DGX platform, where the scale-up interconnect fabric is tightly integrated with liquid cooling, power shelves, busbars, switching infrastructure, and system management. “The rack itself becomes the unit of system design rather than simply a container of independent compute nodes.” Google was one of the first proponents of 1-megawatt racks, outlining the infrastructure changes required to support next-generation AI deployments. “Central to this approach is the introduction of ±400V DC power distribution within the datacenter, leveraging technologies and deployment experience developed for the electric vehicle industry. Higher distribution voltages provide a practical path to delivering megawatt-scale power while reducing current levels, system losses, and cabling complexity,” Khan said. The mechanical infrastructure is evolving in parallel. Open Rack Wide (ORW) extends the capabilities of the Open Compute Project’s rack architecture to support larger, heavier, and significantly more power-dense AI systems. With approximately twice the width of an ORV3 rack and additional structural reinforcement, ORW provides the physical foundation required for next-generation rack-scale AI platforms. AMD adopted this approach with its Helios AI rack architecture, which is based on Meta’s ORW specification submitted to OCP in 2025. The goal is to create a deployable, standardized rack-scale platform optimized for AI data centers. Nvidia packages its latest DGX rack-scale systems within more traditional rack dimensions, demonstrating that multiple implementation approaches are emerging across the industry. While much of the current standardization effort is focused on that shared infrastructure, the choices become more specific once the CSPs decide how to populate those racks with compute, networking, and packaging technologies. That is where 3D-IC design starts to become especially important. “Pretty much all of the big CSPs are looking at 3D-IC design because to hit those kinds of densities, you don’t really need 300 kilowatts, like AMD’s Helios,” noted Daniel Wilkinson, distinguished engineer, machine learning architecture at Synopsys. “I don’t think those are 3D stacked designs yet, but the 1-megawatt ones will be. However, I don’t think just having a bigger dinner-plate system of wafer-type scaling planar works, because that comes with its own serious mechanical challenges and yield challenges and that sort of thing. Just moving the data over that span gets difficult. So they’re all looking at exploring 3D-IC design, looking at taking two compute dies and stacking two dies together. That complicates quite a lot of things and raises quite a lot of challenges, so we will start to see what was maybe four dies on an interposer either become eight dies on an interposer, with four and four, or maybe just two and two. It’s early days to see exactly what these architectures will look like, and different CSPs will take slightly different approaches. If you look at what some of them [have disclosed], like Google’s TPU, it is a big systolic array, like one massive matrix multiplier, and they have two or four of them on a node. Some of the others are taking more of a GPU-like approach, where they’ve got many smaller cores. Those two strategies will lead to different stacking approaches and different stacking constraints. There will be different requirements on the interface IP and the foundation IP, and all of the stuff around the stacking for these different customers. It’s early days, but there are a few ideas as to how it might pan out.” Regardless of the physical form factor, large AI data centers are looking to move from a server-centric design to a rack-centric system architecture. But that raises new issues involving power delivery and power density, especially the amount of power that can be dissipated by both a processor and a rack. “This trend has grown dramatically from about 20 years ago, when we were at about four kilowatts per rack, where today these racks are pushing more than 100 kilowatts of power, and there are some changes in the way that power is delivered,” noted Steven Woo, fellow and distinguished inventor at Rambus. “Rather than having 12-volt supplies, the industry is moving to 48 volts and looking to move well beyond that, potentially to 480 or even 800 volts. The reason why you do this is that at a higher voltage, it allows a lower amount of current to be used to provide the same amount of power, and it turns out that if you kept the voltage at 12 volts, you would need lots more current. But in the existing infrastructure in data centers, the wires are not thick enough to support that much higher level of current. They would melt. And despite the expense of having the copper wiring in there, it’s much cheaper and easier to increase the voltage.” What is the best path forward? Higher voltage only addresses one side of the equation. As rack power climbs, the industry also must decide whether ever-denser centralized infrastructure is the most practical scaling path, or whether efficiency gains will come from changing where and how processing is done. Steve Roddy, chief marketing officer at Quadric, said that while advancing the state of the art in data center power distribution and cooling can undoubtedly deliver increased throughput and performance, it is worth asking whether this extreme engineering is the best way to solve a compute challenge. “Our industry has a decades-long history of oscillating between chasing general-purpose compute and then pivoting to more application-specific compute that is less general yet far more energy- and cost-efficient,” Roddy said. “Similarly, the industry over the past 60 years has seen waves of centralized compute (mainframes in the 1960s), followed by waves of distributed compute growth (the PC era of the 1980s and 1990s), followed yet again by the cloud and today’s hyperscalers. If both of those waves intersect in the coming years — more specialized silicon and a pivot back to distributed compute — then the monumental scaling challenge of the AI build-out will see not just continued scaling of generalized GPU compute in data centers, but also more and more special-purpose chips designed for efficient data center inference or training, lessening the power density of chips in the data center with more efficient devices delivering higher inference per watt. And the need to build endless oceans of warehouses filled with racks can be lessened by deploying AI token generation capability into tens of millions of homes and businesses – generation capability that works in tandem with centralized models running on centralized farms of servers.” Optical adds another piece of the alternative puzzle. “As a company, we’re betting against the 1-megawatt rack getting widespread traction,” said Vishal Chandrasekar, director of product management at Ayar Labs. “Why are people going toward those sorts of power? It is because they’re trying to cram more GPUs within a single rack, keep the distance between the GPUs small, and stick with copper. What we’re saying is, ‘Okay, you don’t want to go to optics for whatever reason. Maybe you’re concerned about reliability, cost, or whatever it may be. Instead of extending in the distance direction, you’re expanding in the thermal density direction. We’re trying to go from 200- to 600-kilowatt to 1-megawatt racks. Where do we go from there? Even if you get to 600kW, are you going to go to 1, 2, 3, or 5 megawatts?’ It’s not a realistic scaling vector because every single time you do that, you’re re-architecting the entire data center in terms of how much power it can hold, and sometimes even how much weight the floor can hold. You look at AMD’s Helios racks. I believe it weighs as much as an elephant. So we don’t see that as a scaling vector.” Optics enables the same number of GPUs to be connected within a scale-up domain to 500,000 or even beyond. “In that case, length is released as a vector, and you don’t care about it anymore,” Chandrasekar said. “You can spread out your GPUs across an entire row, because in data centers, people don’t care about real estate costs so much. Power is the real limiting factor. As long as I can spread out my GPUs, I can still stick with, let’s say, 200-kilowatt racks, but have 10 of them connected via optics. I achieve the same performance as getting a one- or two-megawatt rack, and now I have to re-architect my entire data center. I have to re-architect my entire power delivery and power distribution. That’s a far bigger lift, and I have to do that every two years.” Power delivery is a growing challenge The scaling question becomes even more urgent when viewed from the power-delivery side, where rising rack densities are forcing changes not only in interconnect strategy, but also in the electrical architecture of the data center itself. “The rising data center power needs are driving a huge add-on to data center infrastructure, especially AI data centers that are being built, so we get the news daily that the hyperscalers plan additional data center capacity in the hundreds of megawatts, or like gigawatt scale,” said Christian Hoefling, principal engineer at Infineon Technologies. At the rack level, we talked in the past about maybe below 100-kilowatt power consumption. Today, we are more in the range of 250 kilowatts. In the future, we will talk about up to 1 megawatt in 1 rack. At a GPU level, we’re coming from maybe 100 to 300 amps of peak current in the CPU world. Going over to modern GPUs, we already see today we are in the range of 1,000 amps, 2,000 amps, and we’re expecting these peak currents to hit 10,000 amps per GPU at the end of this decade.” That has big implications for power losses and power distribution. “The power distribution losses in the board itself keep on increasing,” Hoefling said. “Keeping the power delivery resistance the same and doubling the peak current means a factor of four in the power losses, because power losses go with the current squared.” These factors contribute to architectural changes (e.g., 12V to 48/50V to 800V AC, to high‑voltage DC) that are being driven by these rising power needs. “When moving to high‑voltage DC distribution and changing architectures, at the end of the day, it’s all about making it more efficient,” he said. “And an efficiency boost is when you can eliminate [power conversion] steps. Every conversion step that you can eliminate is again a gain in efficiency.” Further, the drive toward higher DC power distribution voltage is simply to meet the requirements that come with higher power in the racks. “If you talk about 1 megawatt of power consumption of a server rack with 800 volts, with a 50-volt busbar, you’re more in the range of like 20,000 amps that you need to support through the busbar,” Hoefling said, noting that data center architectures must change to cut I²R losses. “We see that in the data center and around the data center, the power delivery architectures are changing completely, and this is what requires new technology. And for the end of the decade in gigawatt data centers, we see that there will be a switch from an AC power distribution to a high‑voltage DC power distribution on the complete data center to eliminate some of the power conversion steps, and then to simplify the power delivery to the racks.” Those infrastructure changes are only part of the story. The workload side is shifting, as well, and agentic AI is beginning to reshape power assumptions by turning what used to be intermittent peaks into far more sustained operating conditions. Agentic, long‑running workflows are changing hardware balance and demand, which ties directly into sustained power needs. “With these long‑running agents and all the orchestration they require, the old model of designing for average power and just handling occasional peaks breaks down,” said Sathishkumar Balasubramanian, head of product for EDA AI & Solido at Siemens EDA. “For many of these AI workloads, you have to assume it’s effectively peak power all the time, so power management and analysis become absolutely critical.” AI/agentic workloads are changing the power profile from mostly idle/average‑based design to near-constant peak, forcing much tighter power management and analysis. “Agentic AI creates very irregular, and in some cases truly 24/7 utilization patterns,” Balasubramanian said. “You can’t size the system around an average workload anymore. You have to architect it for sustained peak operation and make sure the entire power and thermal stack can live with that. That means you must make sure that the power management is done very well, and make sure that the power analysis is done and everything is good.” Another added pressure on already extremely complex designs is that AI agents and verification and implementation are more about thermal and 3D‑IC effects that result from the higher switching activity and density driven by AI workloads. Architectures must scale up (more cores, 3D‑IC, etc.), which implicitly raises power density and power‑delivery requirements. “With the pace that development is happening, along with the changes in architectures and new design starts, it’s a good time to be on the verification side,” Balasubramanian said. “Everything is going to be a die stack, which means understanding the physical effect of everything you’re doing. You can have a very high switching bus, but what does it mean in terms of your thermal map?” Agentic AI tends to push designs toward heavier, more continuously engaged compute. “One of the key things in agentic AI is context swapping,” Balasubramanian said. “The biggest thing about the ultimate metric in terms of agentic AI is how fast a processor or system can save, suspend, and restore a complex model because people might be changing back and forth. You don’t want them to wait.” As rack power scales toward 1 megawatt, power increasingly becomes a proxy for compute density. However, just adding more processing elements does not guarantee higher application performance. “The ability of those accelerators to communicate efficiently becomes equally important,” Cadence’s Khan said. “Scale-up interconnects must deliver higher bandwidth, lower latency, improved power efficiency, and broad ecosystem interoperability. AI training and inference workloads are increasingly constrained by memory bandwidth, synchronization overhead, collective communication efficiency, and tail latency. Packing thousands of GPUs into a rack-scale environment only creates value if those GPUs can operate as a coherent computing system. This is where emerging open scale-up standards, such as UALink and other next-generation accelerator interconnect technologies, become increasingly important. They provide the foundation for building large accelerator domains while promoting ecosystem interoperability and innovation.” Shrinking market windows The AI-driven pressures that are driving new power, cooling, and packaging choices are also compressing market windows, leaving companies less time to redesign, validate, and deploy these architectures before the next generation arrives. Market windows are shrinking, and they will continue to shrink, said Manmeet Walia, executive director, product management at Synopsys. “They only have a year’s market window to take that product into production. If they miss the market window, they might as well kill that generation and move on to the next one. From our perspective, we must be even faster. The specs are ratified within a 1.5- to 2-year window. We need to have a full subsystem test chip available by the time the spec is ratified. For example, in the world of memories, there are three HBM vendors. We need to align our test chip, a subsystem test chip that is going to look like the IP that will be going into customers’ SoCs by the time these devices are made available. Let’s say Samsung has HBM5 ready on this date. We need to have our full test chip so that they can plug it in, build the system, and test it out, bearing in mind that our test chip has to be packaged on a co-op substrate with the HBM memory. And that takes many months, which takes many months off the time you have to get that test chip done.” Advanced packaging extends timelines, as well, creating pressure to prepare earlier so that the entire cycle can happen smoothly. The goal is to have the final test vehicle ready and validated by the next deployment stage, which is non-trivial. “These programs are very expensive, running into a billion dollars for an accelerator, so they cannot go wrong on these,” Walia said. “They cannot take the unverified, unvalidated IP. Being ahead of a standard, which is already going at a pace faster than the silicon cycle, is keeping us up at night. It’s not sustainable long-term. At some point, this whole madness needs to end.” Might that look like more reasonable development? “One of the mitigating things for that is models,” said Synopsys’ Wilkinson. “People talk about models always getting bigger, and the biggest models are getting bigger. But the workhorse models that form 70% of the inference workload are probably midsize — a trillion parameters rather than 10 trillion. So, if you look at something like Google’s suite of consumer AI models, there may be a trillion parameters or even smaller. A lot of that workhorse inferencing, which is where the majority of tokens are being generated, has fewer requirements. There’s an element of focus on TCO being very disciplined about all those workhorse considerations, and then you’ve got the huge training runs and the really big models like Claude and the like, which need a somewhat different infrastructure. Even within that whole ecosystem, a lot of the agentic stuff can be smaller models, and sometimes it is, so it’s not like everything’s massive. There are massive things, but you see with some CSPs, they have more than one platform, and they might use Nvidia or AMD for some things, but they’ve also got their own internal infrastructure that is somewhat differently focused.” This is a layered and complicated picture where a lot of the internal programs may be more directed at the workhorse models that they’re running, as opposed to the flagship models that get everybody’s attention. So maybe everything doesn’t run on a 1-megawatt rack. Wilkinson believes there will be more fragmentation in these solutions, whether they’re internal solutions from big hyperscalers or from startups or other players who are trying to enter the market, maybe on-prem at enterprises. “Not everyone wants to get their AI from the cloud,” he said. “An enterprise is probably not going to put a 3-ton, 500-kilowatt rack on its own premises. They will need solutions that work on-prem within the IT room capabilities they have, and for the workloads they have, which might not even be LLMs. It’s a huge market, and there’s no possibility that any one player can corner all of the workloads and use cases. The way the madness ends is things settling out into segments and specializations with different equipment for different use cases, some of which won’t be on quite such a compressed timescale or bleeding-edge technology.” That fragmentation, however, does not make the integration challenge go away. If anything, it makes coordination across the full stack even more important. Cadence’s Khan noted that the transition to 1-megawatt racks is not fundamentally a rack problem. It’s an ecosystem integration problem. “Success requires coordinated innovation across power delivery, cooling, networking, interconnects, memory systems, optics, software, and data center infrastructure,” he said. “The winners in the AI era will not be the companies that optimize a single component in isolation. They will be the companies that optimize the entire path, from grid to chip, chip to chiller, and accelerator to accelerator, treating the rack, the network, and ultimately the data center as one integrated AI system.” Leave a Reply