DevOps6 min read

Disaster Recovery & Backup

Design and test disaster recovery plans - defining RTO/RPO, automated backups, multi-region failover, and regular DR drills that actually work under pressure.

Disaster RecoveryBackupRTORPOHigh Availability

skill: Disaster Recovery & Backup category: DevOps tags: [Disaster Recovery, Backup, RTO, RPO, High Availability] readingTime: 6

A disaster recovery plan you've never tested is a hope, not a plan. Every DR plan has a gap between theory and reality; testing reveals the gap before the incident does. Measure RTO and RPO concretely, automate everything possible, and run a DR drill at least annually - or after every major architecture change.

Workflow

Define RTO and RPO requirementsRTO (Recovery Time Objective) and RPO (Recovery Point Objective) define what's acceptable to the business - they drive every DR technical decision.
Automate database backupsManual backups are forgotten, misconfigured, and untested. Automated, encrypted, offsite backups with restore testing are the standard.
Set up database replicationReplication provides a hot standby that can take over in minutes - meeting aggressive RTO requirements without the complexity of multi-region.
Design a multi-region failoverSingle-region failure (datacenter fire, region outage) takes down a single-AZ setup. Multi-region provides geographic redundancy.
Test DR plans regularlyAn untested DR plan is theater. The only way to know RTO and RPO are achievable is to measure them in a drill - before the actual disaster.

Output Format

DR plan: RTO/RPO requirements, automated backups with restore testing, replication, multi-region failover, and annual DR drills.

DR Plan:
  Tier 1 services: RTO 1h, RPO 15min
    → RDS Multi-AZ (auto-failover)
    → Hourly snapshots + WAL archiving
    → Route 53 health check failover

  Backups: daily full + hourly incremental → S3 (encrypted)
  Restore test: automated in CI weekly
  DR drill: biannual, recorded in runbook

Examples

Input

Output

Edge Cases & Gotchas

Backup of application code - Database backups alone aren't enough - application code, config files, secrets references, and infrastructure-as-code must also be backed up. Git repositories + IaC state in remote storage cover this.
Secret rotation during recovery - DR environments may use different credentials than production. Automate secret injection from a secrets manager (AWS Secrets Manager, HashiCorp Vault) rather than embedding credentials in backups.
DNS TTL and failover speed - Low DNS TTL (60s) enables fast failover; high TTL (24h) caches the old IP. Set TTL to 60s for critical records - this allows Route 53 to fail over within 2-5 minutes.
Partial failures - Not all failures are total outages - a database primary failing while the application is running creates an ambiguous state. Implement circuit breakers to fail gracefully rather than hanging indefinitely.

When to Break These Rules

Use a managed database service (RDS, Cloud SQL, PlanetScale) instead of self-managed PostgreSQL for built-in backup, replication, and failover - eliminates most DR complexity.
Use Velero for Kubernetes backup instead of scripting - it handles PersistentVolumes, namespace backups, and cross-cluster restore with a consistent API.

Reference Files

https://aws.amazon.com/disaster-recovery/
https://sre.google/workbook/table-of-contents/