Multi-Cluster databases on Kubernetes: Architecture and deployment

Introduction Running a database on Kubernetes is well understood. Running one that survives a complete regional failure, a corrupted control plane, or a severed network requires a fault-resistant architecture. This post walks through how to build a multi-cluster MongoDB deployment on Kubernetes that can withstand those failures. We will use the Percona Operator for MongoDB, an open-source Apache 2.0 Licensed Kubernetes Operator, as our example. For relational workloads, similar patterns are available through projects like Vitess and CloudNativePG. If you are new to multi-cluster MongoDB on Kubernetes, Ivan Groenewold’s post is a great place to start. It walks you through a complete multi-cluster setup you can follow hands-on. Single Point of Failure: The risk of a single-cluster setup. Standard Kubernetes excels at self-healing within a single cluster, automatically recovering from crashed Pods or failed Nodes. However, it has no built-in mechanism to handle failures at the cluster level itself: a regional outage, a corrupted control plane, or a network partition can take down your entire database with no automatic recovery path. Distributing MongoDB nodes across separate Kubernetes clusters addresses this gap and enables three core scenarios: - Disaster Recovery (DR): If an entire region goes down, nodes in a secondary cluster already hold a full copy of the data and enough votes to elect a new Primary and resume writes automatically. - Live Migrations: Running nodes across two clusters lets you shift application traffic gradually between environments, for example, during a cloud provider migration, without taking the database offline. Note that coordinating a clean cutover still requires careful application-level planning around connection strings and write consistency. - Maintenance without Stopping: You can drain and upgrade one cluster entirely while the database continues accepting writes through the nodes in the remaining cluster, with no scheduled downtime. Architecture Overview To manage the complexity of multi-cluster deployments, the architecture divides clusters into two roles, ensuring that Kubernetes-level operations (handled by the Operator) and database-level operations (handled by MongoDB) do not interfere with each other. All nodes, regardless of which cluster they run in, belong to a single MongoDB replica set, which is what enables cross-cluster voting and leader elections. 1. Defining Site Roles Each cluster is assigned to one role to prevent conflicts between independent Kubernetes control planes: - Main Site: The primary cluster, fully managed by the Operator. It holds the Primary node and handles application writes. The Operator here is responsible for provisioning TLS certificates, user credentials, and replica set configuration. - Replica Site: The Operator runs with unmanaged mode to true, which means it does not generate certificates or user credentials, and does not attempt to initialize a new replica set. Instead, the user needs to copy TLS secrets and credentials from the Main Site, allowing the Replica Site nodes to authenticate and join the existing replica set. This prevents the problem where two independent operators attempt to control the same database simultaneously. 2. Connecting Clusters with the MCS API Even though each Kubernetes cluster runs an independent control plane, the MongoDB replica set spans across all of them. The Kubernetes Multi-Cluster Services API (MCS API) makes this possible. When multiCluster.enabled: true is set, the Operator creates ServiceExport and ServiceImport resources so nodes in different clusters can discover and communicate with each other via a shared DNS zone (svc.clusterset.local ). Note that the MCS API is not included in a standard Kubernetes installation and requires a separate implementation such as Submariner, Cilium ClusterMesh, or a managed provider solution like GKE’s native MCS. To enable multi-cluster service discovery with the Percona Operator for MongoDB, set multiCluster.enabled: true in your cr.yaml : apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDB metadata: name: main-cluster spec: crVersion: 1.23.0 image: perconalab/percona-server-mongodb-operator:main-mongod8.0 imagePullPolicy: Always updateStrategy: SmartUpdate multiCluster: enabled: true # <--- Enable multi-cluster service discovery DNSSuffix: svc.clusterset.local # <--- The Global DNS standard After applying the configuration, verify that the Operator created the ServiceExport resources. Note that it takes approximately five minutes for resources to sync across the fleet. This is how it looks: kubectl get serviceexport NAME AGE main-cluster-rs0 6m main-cluster-rs0-0 6m main-cluster-rs0-1 5m The Main Site (Cluster A) runs the Primary node and is fully managed by the Operator. The Replica Site (Cluster B) holds Secondary nodes in unmanaged mode. ServiceExport and ServiceImport resources bridge the two clusters via a shared svc.clusterset.local DNS zone, enabling cross-cluster replica set membership. Image 1: Multi-cluster MongoDB deployment using the Kubernetes MCS API. How to Design for High Availability Before choosing how to distribute clusters, it helps to understand how MongoDB itself decides which node is in charge. A MongoDB replica set requires a strict majority of its voting members to elect a Primary and accept writes. With four voting members spread equally across two locations, neither side can reach a majority if the network between them is severed, and both sides stop accepting writes, even if every server is healthy. The solution is a fifth member, placed in a third location. With 5 total votes, one side reaches 3 votes; a majority, and elects a new Primary. This is the 2+2+1 pattern. Image 2: The following image is in a Normal State with no Network Partition Image 3: After Network Partition Have an odd number of MongoDB voting members, placed in different locations, so that if one location or network link fails, one side can still have enough votes to keep/elect a Primary. With the Percona Operator for MongoDB, the 2+2+1 pattern is configured directly in cr.yaml. Start by defining the total number of data-bearing replica set members across both sites: replsets: - name: rs0 size: 2 # 2 nodes in Main Site + 2 nodes in Replica Site expose: enabled: true type: ClusterIP … Then add the extra member as a separate entry under the same externalNodes section: externalNodes: - host: third-site-rs0-4.psmdb.svc.clusterset.local votes: 1 priority: 1 Deployment Workflow The following steps deploy the 2+2+1 pattern across two Kubernetes clusters using the Main and Replica site roles described above. - Connect the Networks: Set up cross-cluster network connectivity using Cilium ClusterMesh, Submariner, or your cloud provider’s native solution (such as GKE Fleet). This is a prerequisite; the MCS API requires an implementation to be installed before the Operator can create functional ServiceExport andServiceImport resources. - Deploy the Main Site: Deploy the Operator and apply your cr.yaml on the primary cluster. Ensure replica set pods are exposed withtype: ClusterIP , this is required by the MCS API since nodes communicate internally across clusters. See the Main Site configuration guide. - Copy the Secrets: The Main and Replica sites must share the same TLS certificates and user credentials to communicate. Run kubectl get secrets on the Main Site to confirm the exact names in your deployment. - Deploy the Replica Sites: Start the remote clusters, but remember to set their operators to “unmanaged” mode. This tells the nodes to join the existing database instead of trying to create a new one. apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDB metadata: name: replica-cluster spec: # ---------------------------------------------------- # This tells the Operator NOT to start a new database, # but to join the existing one to prevent split-brain. # ---------------------------------------------------- unmanaged: true multiCluster: enabled: true DNSSuffix: svc.clusterset.local - Finalize the Replica Set: Update cr-main.yaml on the Main Site to add the Replica Site nodes asexternalNodes . Two nodes are added as voting members with lower priority, and the third as a non-voting member, this prevents split-brain if the Replica Site loses connectivity. replsets: - name: rs0 size: 2 externalNodes: - host: replica-cluster-rs0-0.psmdb.svc.clusterset.local votes: 1 priority: 1 - host: replica-cluster-rs0-1.psmdb.svc.clusterset.local votes: 0 priority: 0 Then apply: kubectl apply -f deploy/cr-main.yaml. Repeat this step on the Replica Site, adding the Main Site nodes as externalNodes in cr-replica.yaml . Please check the interconnect guide if you want to see a full configuration. replsets: - name: rs0 size: 2 externalNodes: - host: main-cluster-rs0-0.psmdb.svc.clusterset.local votes: 1 priority: 1 - host: main-cluster-rs0-1.psmdb.svc.clusterset.local votes: 0 priority: 0 Once these steps are complete, the replica set spans both clusters and is ready for the failover behavior described in the next section. Failover Behavior: The Election Process MongoDB’s replica set protocol includes built-in failure detection and automatic leader election. If the Primary node goes offline, whether a single pod crashes or an entire region is down, the remaining nodes automatically elect a new Primary without any operator or human intervention. Let’s look back at our 2+2+1 setup. To stay online, the database needs a strict majority (3 out of 5 votes). If Cluster A completely fails, the two nodes in Cluster B plus the node at Cluster C give us exactly the 3 votes we need to choose a new Primary. Under the default configuration, the median time before a cluster elects a new primary does not typically exceed 12 seconds; this includes the time to detect the failure and complete the election. In a multi-region setup, cross-cluster network latency may extend this window, so your application connection logic should include retryable writes to handle the brief period during which no Primary is available. The electionTimeoutMillis setting can be tuned if your latency requirements demand faster detection. Architecture After Failover After a regional failure, your database is still working, but it enters a “degraded” state. This means it is functioning, but with fewer resources than normal. - The New Primary: A node in Cluster B is promoted to be the new leader and starts accepting new data. - Less Redundancy: Your database is now running with the exact minimum number of nodes needed for a majority. In our 2+2+1 setup, you are surviving on just those 3 remaining members. - Reduced Fault Tolerance: Even though your application survived the crash of Cluster A, your safety margin is completely gone. If you lose just one more node in Cluster B or the node at Cluster C, the database will lose its majority and lock into “read-only” mode. Image 4: Failover in a multi-cluster MongoDB deployment using the Kubernetes MCS API The replica set detected the failure, held an election, and promoted a new Primary, automatically, without human intervention. That is the 2+2+1 pattern working as designed. In a follow-up post, we will put this architecture under deliberate pressure with chaos engineering experiments to measure exactly how it holds under real failure conditions. This blog post was reviewed by Chetan Shivashankar, Technical Lead, Kubernetes at Percona. Here are some resources you can review to learn more about the topic.