Technical blueprint, not a client delivery

Planning EKS recovery without two active writers

A standby only helps if the team can recover data and resume service without creating a second writer.

This warm-standby blueprint combines a scaled-down EKS environment with replicated data and images, independent backups, and a deliberate sequence for fencing, promotion, scale-up and traffic shift.

The design would give operators a rehearsable failover sequence while keeping replication distinct from backup recovery.

Recovery designs fail when a standby lacks images, secrets or capacity, when replication is mistaken for backup, or when the runbook depends on the failed region. The recovery objective belongs to the workload owner and must be measured in drills, not inferred from a diagram.

A warm-standby EKS design that distinguishes replication from backup and uses a human-controlled failover runbook instead of promising untested RPO or RTO.

Prepared standby, deliberate recovery

Replication keeps selected dependencies current, backup protects from logical damage, and a runbook controls promotion.

No recovery target is asserted here; measure detection, decision and restoration in drills.

Normal: Serve in primary: The primary EKS cluster serves traffic and writes to the primary database. → Replicate and back up: Data and artifacts replicate, while independent backup recovery points are copied to the recovery region. → Keep standby current: Argo CD deploys the same versioned manifests to the warm cluster.
Failover: Detect and decide: Health signals raise an incident; an operator confirms failover and starts the recovery-region runbook. → Fence and promote: Prevent primary writes before promoting the Aurora secondary. → Scale and shift: Raise node and workload capacity, check dependencies, then enable recovery traffic through ARC controls.
Primary region: The active EKS workload serves requests and writes to the primary data services. Amazon EKS, Aurora primary, S3, ECR
Replication: Database, object, image and secret copies keep the recovery environment supplied, with service-specific lag and key requirements. Aurora Global Database, S3 Replication, ECR replication, Secrets Manager replicas
Warm standby: A running, scaled-down EKS cluster receives the same manifests from Git and scales during a planned recovery. Amazon EKS, Argo CD, Terraform, Recovery capacity
Independent recovery: Cross-region backup recovery points protect against logical damage that replication also propagates. AWS Backup, Backup vault, Vault Lock, Point-in-time recovery
Failover control: Health signals inform an operator; a runbook fences, promotes and shifts traffic with recovery-region dependencies available. Route 53 health checks, Systems Manager Automation, Route 53 ARC, Runbook
Why are replicated copies not enough? A delete or corrupted update can replicate to the secondary before it is noticed. Maintain independent, cross-region backup recovery points in addition to data replication. Regional failover and recovery from logical damage have separate procedures and evidence.
How does the design avoid split brain? Promoting the secondary while the primary can still accept writes risks two writers. Fence or stop primary writes before promoting the secondary and only then shift routing. Failover is ordered and operator-controlled instead of a blind health-check reaction.
What RPO and RTO can be promised? The architecture alone does not measure replication lag, detection time, decision time or recovery readiness. Set targets with workload owners and record results from planned and unplanned drills. Recovery objectives are evidence-based and include human decision time, not just service promotion time.

The operational failure modes this design addresses.

Warm standby costs money before it serves production. Asynchronous replication has nonzero lag, while automatic failover can create unsafe state transitions. Short DNS TTLs do not force every client to stop caching answers; client behavior and recovery-region quotas also belong in the drill.

Recoverability: Keep the recovery region deployable. A running but scaled-down cluster, replicated images and secrets, and infrastructure defined from the same source reduce the number of dependencies that must be rebuilt during an incident.
Data protection: Separate replication from recoverable copies. Replication helps keep a secondary current but can copy logical damage. Cross-region backup recovery points provide a distinct recovery path.
Controlled failover: Prevent two writers and sequence dependencies. Fence the primary before promoting a database, scale compute and workloads together, then shift traffic through the controlled routing mechanism.
A running EKS cluster in the recovery region receives the same manifests and can be scaled during recovery.
Database, object, container-image and secret copies are planned with regional lag and encryption-key requirements.
Cross-region recovery points protect against logical damage that would also propagate through replication.
Detection, human decision, primary fencing, database promotion, capacity scale-up and traffic shift are separate steps.

How the system is intended to work.

A warm-standby EKS design that distinguishes replication from backup and uses a human-controlled failover runbook instead of promising untested RPO or RTO.

Warm compute standby: A running EKS cluster in the recovery region receives the same manifests and can be scaled during recovery.
Replicated dependencies: Database, object, container-image and secret copies are planned with regional lag and encryption-key requirements.
Independent backups: Cross-region recovery points protect against logical damage that would also propagate through replication.
Runbook-led recovery: Detection, human decision, primary fencing, database promotion, capacity scale-up and traffic shift are separate steps.
Primary region: The active EKS workload serves requests and writes to the primary data services. Amazon EKS Aurora primary S3 ECR
Replication: Database, object, image and secret copies keep the recovery environment supplied, with service-specific lag and key requirements. Aurora Global Database S3 Replication ECR replication Secrets Manager replicas
Warm standby: A running, scaled-down EKS cluster receives the same manifests from Git and scales during a planned recovery. Amazon EKS Argo CD Terraform Recovery capacity
Independent recovery: Cross-region backup recovery points protect against logical damage that replication also propagates. AWS Backup Backup vault Vault Lock Point-in-time recovery
Failover control: Health signals inform an operator; a runbook fences, promotes and shifts traffic with recovery-region dependencies available. Route 53 health checks Systems Manager Automation Route 53 ARC Runbook

Test the boundaries before relying on the design.

The following checks define evidence to collect before implementation is accepted. None is represented as already run.

Warm compute standby: A running EKS cluster in the recovery region receives the same manifests and can be scaled during recovery.
Replicated dependencies: Database, object, container-image and secret copies are planned with regional lag and encryption-key requirements.
Independent backups: Cross-region recovery points protect against logical damage that would also propagate through replication.
Runbook-led recovery: Detection, human decision, primary fencing, database promotion, capacity scale-up and traffic shift are separate steps.
Failover and failback: Run a planned switchover and the separate resynchronization and failback procedure; record lag and elapsed times.
Primary isolation: Verify the recovery runbook can proceed when primary-region APIs are unavailable and that a reachable primary is fenced before promotion.
Backup restore: Restore a deliberately selected object or database recovery point from the recovery-region backup, not from its replica.
Standby readiness: Confirm images, secrets, keys, quotas and dependencies work in the recovery region before a traffic shift.
Routing safety: Verify ARC safety rules and runbook gates prevent an invalid dual-region routing state.
No measured performance, cost, recovery or compliance outcome is published for this project.

A design that makes operational boundaries reviewable.

The intended value is a recovery process that can be rehearsed and measured without assuming replicated data is always recoverable. RPO and RTO remain unproven until drills establish them. No measured performance, cost, recovery or compliance outcome is published for this project.

Rehearse a controlled promotion: A runbook would fence the primary, promote data and move traffic in a defined order.
Keep an independent recovery point: Cross-region backups would provide an alternative when replication has copied logical damage.

Review the constraints before selecting the implementation.

Start with workload limits, owners and failure behavior. Then turn the validation plan into environment-specific tests and acceptance criteria.