Insights · Platform Engineering

Disaster recovery by design: RTO and RPO you can prove

HA/DRVeleroMulti-site

Key takeaway

Pick RTO and RPO per workload, choose the recovery strategy that matches the numbers, and prove it with timed failover tests.

The two numbers that define disaster recovery have precise, standards-track definitions. NIST’s CSRC glossary, carrying SP 800-34 Rev. 1 — the Contingency Planning Guide — defines RTO as “the overall length of time an information system’s components can be in the recovery phase before negatively impacting the organization’s mission or mission/business processes,” and RPO as “the point in time to which data must be recovered after an outage.” AWS’s Well-Architected guidance adds the distinction practitioners trip on: RTO is a maximum target for a single event; MTTR is a mean across many. Revenue-critical, safety-critical, and internal workloads do not deserve the same numbers — the whole discipline starts with tiering the estate and setting objectives per tier.

The four recovery strategies, priced in RTO/RPO

AWS’s reliability guidance names the four canonical strategies — “listed in increasing order of cost and complexity, and decreasing order of RTO and RPO”:

  1. Backup and restore — RPO in hours (point-in-time recovery can cut it to minutes), RTO up to 24 hours. Cheapest; the floor for everything.
  2. Pilot light — core data stores replicate continuously; compute is deployed on demand. RPO in minutes, RTO in tens of minutes.
  3. Warm standby — a scaled-down full copy always running. RPO in seconds, RTO in minutes; scale it up during failover.
  4. Multi-site active-active — RPO near zero, RTO potentially zero. The most expensive; justified only where downtime costs more than the duplicate estate.

The guidance is refreshingly honest about the economics: “you should avoid implementing a strategy that is more stringent than it needs to be.” This is exactly the shape of a decision that belongs in an ADR — per workload tier: chosen strategy, the objectives it must meet, the accepted residual risk, and the cost. On the E-Tax programme, the DR strategy for the taxpayer-facing tier was recorded before the first failover runbook was written, which made every later conversation about DR a constraint check instead of a renegotiation.

Backups are not recoverability

The most load-bearing sentence in the Google SRE book’s data integrity chapter is also the most ignored: “replication and redundancy are not recoverability.” Replication faithfully propagates corruption and deletion to every copy, on every continent, at line rate. A recovery strategy that assumes the replicas are clean will fail exactly when DR matters — ransomware and logic bugs both write replicated data.

The chapter’s operating philosophy is restore-first: “no one really wants to make backups; what people really want are restores.” That inverts the design: retention tiers are defined by restore objectives (Google’s tiers ran from minutes-fresh backups restorable quickly, through weeks, to nearline/archive storage — with a 2012 restore of 1.5 PB from roughly 5,000 tapes taking about a week as the cautionary tale of off-clip recovery). For most estates the practice is:

  • Frequent short-retention backups for rapid recovery from the ordinary disaster (bad deploy, dropped table).
  • Isolated, access-controlled copies — credentials for the recovery copies must not live in the same identity domain as production, or the ransomware operator deletes both.
  • Restore drills on a schedule, with measured timings — a backup that has never been restored is a rumor.

Prove it: failover tests, game days, chaos

The AWS best practice for testing DR implementation states the thesis flatly: “regularly test failover to your recovery site to verify that it operates properly and that RTO and RPO are met” — and attaches the institutional memory: “our experience has shown that the only error recovery that works is the path you test frequently.” The failure to establish this testing is rated a High risk. The anti-pattern they call out — never exercising failover in production — is how organizations own a DR plan that works in the binder and dies in the datacenter.

The test cadence that works:

  1. Restore drills — monthly, per tier: restore the backup into an isolated environment, validate integrity, record the elapsed time against RPO/RTO.
  2. Failover game days — quarterly: actually switch traffic to the recovery site (or a drill instance of it), let on-call run the runbook, measure everything, then fail back. The failback is half the test; teams that skip it discover their second disaster on return day.
  3. Chaos experiments — continuous, controlled: “the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production” (Basiri et al., IEEE Software 2016). Terminating instances and severing dependencies on purpose — the practice Netflix normalized with Chaos Monkey — converts resilience from an assumption into a measured property. The 2024 multi-vocal literature review of chaos engineering (arXiv) shows the practice has matured into a defined field with platforms and taxonomies, not just folklore.

Azure’s reliability framing makes the responsibility split explicit: the platform supplies replication and failover building blocks, but “your design and operational choices determine” whether the workload is actually recoverable within defined time and data-loss limits. The cloud does not sell resilience; it sells the parts. Industrial research reaches the same conclusion from the other direction — even hyperscale database failover systems treat customer-chosen RPO and consistency as explicit design parameters to be engineered around (Rowe et al., arXiv 2025).

Runbooks and DR-as-code

A failover executed by reading a wiki at 4 a.m. is a failover that will not meet RTO. Two disciplines close the gap:

  • Runbooks are rehearsed artifacts — versioned beside the infrastructure, updated after every game day with what actually broke. The measure of a runbook is whether someone who wasn’t in the design meeting can execute it under stress.
  • DR is code — the recovery environment is provisioned by the same infrastructure-as-code pipelines as production, so it can be rebuilt from source at 4 a.m. without tribal knowledge, and policy gates (policy as code) verify that backup and replication settings haven’t drifted. If the recovery topology can only be assembled by hand, it is already out of date.
Rule of thumb

If your last DR test isn’t in the incident log with measured timings, your RTO and RPO are fiction — and an auditor (or an attacker) will find out before you do.

Test your recovery, not your plan

Sources