Where should AI workloads run? A sovereign and sensible approach

Opinions on AI range from transformative optimism to deep skepticism, but one thing is clear: AI is becoming an increasingly important part of enterprise technology strategies. Feel free to pick whichever you like. But whatever you picked, the fact remains that AI is not going away and will be useful especially for enterprises and for coding. So, quo vadis, AI? Which AI model you prefer can be endlessly debated. This article explores a different question(s): What should AI workloads run on… and what do you run AI workloads on? On one level, this debate is close to settled: Kubernetes has emerged as a common foundation for AI infrastructure because of its resource management, automation, portability and operational consistency. On another level, the answer remains elusive and poses more questions: where will the models run? Will companies consume AI as an external service? Will they rent raw capacity and run whichever models they choose? Or, as AI workloads become more strategic, expensive and data-sensitive, will they move into private clouds, colocated environments, sovereign infrastructure or on-premises data centers? Weighing the options Currently many proprietary frontier models continue to outperform open-weight alternatives in areas such as reasoning and general-purpose capability. However, not every AI workload needs to be handled by bleeding-edge, token-burning models. Routine, repetitive, narrowly defined tasks can often be handled by open-weight models, older model versions, or even consumer-grade hardware. If those tasks involve sensitive data, internal processes or highly regulated operations, on-premises or private cloud environments may become the more sensible option, possibly even the only viable one. Control over data, compliance and stable long-term planning are strong arguments for controlling every part of the stack yourself. Even outsourced hosting in ‘bring your own model’ setups can serve a useful purpose. Testing new configurations or non-sensitive test environments are excellent candidates for outsourcing hosting while maintaining control over the model. When managed carefully, this approach can also be used to handle peak demand or offload tasks that can be easily compartmentalized. The matter of cost The writing is on the wall: AI companies or divisions can’t operate at a loss forever. Infrastructure investments are sky-high with data centers for running AI workloads at the forefront of spending. To fund these investments, revenue generation will need to increase significantly. As a result, increased prices for consumers, private or corporate, of AI workloads are inevitable. So, is that the end of AI as a service? Probably not. But it might spell the end of “just buy a subscription” thinking. Of course, in terms of sovereignty, that thinking was never sensible or viable in the first place. But pretending that cost isn’t a huge factor even for sovereignty considerations would be kidding yourself. Sovereignty through the lens of compliance and regulations Regarding sovereignty, at KubeOps we work with public-sector organizations and critical infrastructure applications in Germany. That means we operate in a highly regulated field where sovereignty must be reflected in our software and in our processes. Since there is no single, universally accepted definition of digital sovereignty, we treat it as an ongoing process built around five elements: - Operational autonomy refers to the ability to control and manage all elements of a system. - Compliance is a prerequisite for digital sovereignty because, without legal certainty—for example, regarding data storage—no system can operate sustainably. - Auditability is essential to ensure that security and sovereignty are verifiably present. - Portability is essential, as technical, contractual, or organizational dependencies fundamentally restrict operational autonomy. It also guards against sudden price increases. - Resilience is part of digital sovereignty, as a system cannot be sovereign if it lacks redundancies, recovery mechanisms, or robust processes in the event of a crisis. Next steps With all the abstract pondering done, the question remains: How to move forward? A good starting point is that many of the requirements for usual workloads did not change substantially with AI workloads entering the field. Monitoring, backup capabilities, lifecycle management and observability are still key concerns. So, before moving serious AI workloads onto any platform, organizations should perform an AI readiness check, so that’s one of the steps we are taking. That means looking at accelerator capacity, storage performance, data locality, network isolation, identity integration, monitoring, backup, recovery, software supply, vulnerability management and policy enforcement. Without that groundwork, the platform may be sovereign in name but fragile in practice. Building for an uncertain future Our answer is to build for choice. We expect AI workloads to land in several places at once and that is why Kubernetes and open source projects will be a vital component of the emerging, highly dynamic landscape. The AI landscape itself is missing many of the standards we take for granted elsewhere. Even something as simple as project knowledge files can differ from one model environment to another. In that landscape, the infrastructure layer needs to be even more portable and adaptable, not less. In a rapidly evolving AI landscape, Kubernetes’ portability and operational consistency can help organizations adapt without rebuilding their platforms for every new model, provider or deployment pattern. A big takeaway for us is that the key is not to guess the perfect destination today, but to avoid building a dead end. Workloads should be portable. Operations should be reproducible. Security should be enforceable. Migration should be realistic. Costs should be visible. And the platform should remain useful whether the next workload lands on premises, in a private environment, at the edge or across multiple clouds. The current AI landscape is full of promises about what tomorrow might bring, but long-term infrastructure decisions need more than optimism. Costs will change. Model capabilities will change. Regulations will change. The only sovereign and sensible response is to build platforms that can change with them. Because if you don’t, you are likely to get caught out by something, whether that is a new data-location requirement, costs exploding with one vendor or some unforeseen issues emerging in the near future. Kubernetes and the wider CNCF ecosystem provide a practical foundation for this approach. Portable workloads, reproducible operations, policy enforcement and deployment flexibility allow organizations to adapt as technologies, regulations and business requirements evolve. Rather than optimizing for a single deployment model, many organizations are likely to benefit from platforms that preserve choice and make it easier to move workloads across environments as circumstances change.