Database Disaster Recovery Runbook: RTO, RPO, and PITR Procedures

A database disaster-recovery runbook is an executable decision and recovery procedure, not a promise that a backup exists. It assigns authority, distinguishes infrastructure failure from logical corruption, fences unsafe writes, restores or fails over by an approved path, validates business data, and records whether the service actually met its recovery objectives. This template uses Amazon RDS for PostgreSQL point-in-time recovery (PITR) as its concrete restore path; adapt commands and validation to each engine and platform you operate.

Define Objectives and Evidence Before an Incident

Recovery time objective (RTO) is the maximum business-approved time to restore the service. Measure it from the agreed incident start until the application is healthy for users, including detection, authorization, database recovery, networking, secrets, application reconnection, validation, and backlog processing. Recovery point objective (RPO) is the maximum acceptable loss of committed business data. Measure the achieved recovery point from verified records or transaction evidence on the recovered system, not from a vendor's typical replication delay.

Keep current owners, account and Region identifiers, dependency maps, escalation contacts, break-glass access, infrastructure definitions, and a validation query pack beside the runbook. Store credentials in the platform's secret manager, never in this document. The deeper PostgreSQL backup and recovery guide explains backup architecture; this page focuses on incident execution.

First Ten Minutes: Declare, Scope, and Fence

1. Declare the incident. Name the incident commander, database operator, application validator, communications lead, and decision recorder. Start a UTC timeline.
2. Confirm the symptom from two paths. Check application failures and database or provider state. Preserve relevant logs and metrics before automated retention removes them.
3. Stop harmful automation. Pause schema deploys, backup pruning, batch writers, failover loops, and repair jobs that could change evidence or create competing writers.
4. Fence writes when histories could diverge. Put the application in a documented read-only or unavailable mode, stop background consumers, revoke or block writer access where the topology permits, and verify the old endpoint is not receiving writes.
5. Record the suspected bad-data boundary. For deletion or corruption, capture the earliest known bad event, the last known good event, time-zone assumptions, transaction identifiers, and upstream events that might be replayed.

Do not declare an RDS instance failed from CPU or connection metrics alone. A credential rotation, exhausted pool, network policy, bad migration, or application release can look like database failure. A database restore does not repair those causes.

Choose Failover, Restore, or Repair

Failover and PITR solve different problems. Failing over after a destructive statement usually promotes the same bad data. Restoring after a transient host failure can increase downtime without benefit.

RDS for PostgreSQL PITR Procedure

Amazon RDS PITR creates a new DB instance and leaves the source unchanged. Before starting, inspect EarliestRestorableTime and LatestRestorableTime, confirm the chosen UTC timestamp is within the retention window, and capture the source instance's engine version, class, storage, subnet group, security groups, parameter group, option group, encryption key, tags, monitoring, log exports, and secret dependencies. AWS documents that a restore can otherwise receive default networking and parameter settings, so configuration parity is part of recovery.

aws rds describe-db-instances \
  --db-instance-identifier prod-postgres \
  --query 'DBInstances[0].{Earliest:EarliestRestorableTime,Latest:LatestRestorableTime}'

aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier prod-postgres \
  --target-db-instance-identifier prod-postgres-dr-20260801 \
  --restore-time 2026-08-01T04:17:00Z

aws rds wait db-instance-available \
  --db-instance-identifier prod-postgres-dr-20260801

Use reviewed infrastructure code or explicit restore options to reproduce the intended private subnet, security groups, parameter group, instance capacity, storage, encryption, monitoring, and log exports. Do not make the recovered endpoint public for convenience. A new instance can report available while restored storage continues lazy loading, so include representative read latency in validation and warm only approved paths.

Validate Before Moving Traffic

1. Identity and configuration: verify engine version, database names, extensions, roles, parameter group, time zone, TLS requirements, and network reachability from the recovery application.
2. Recovery point: query immutable business identifiers around the chosen boundary. Confirm the last expected good change exists and the first known bad change does not.
3. Integrity: run application-specific invariants such as balanced ledger totals, parent-child counts, uniqueness checks, queue offsets, and recent object references. Row counts alone are not proof of correctness.
4. Read path: run bounded representative queries and compare critical aggregates with independent records. Investigate unexpected errors or plan regressions.
5. Write path: while isolated, execute an idempotent synthetic transaction, read it through the application, and roll it back or remove it through an audited path.
6. Operations: verify backups, alerts, logs, secret access, connection limits, scheduled jobs, replicas, and dashboards are configured for the new identifier.

The application owner and data owner must sign off on these results. If validation fails, keep traffic fenced and select another restore point or recovery method; do not mutate the only promising recovery copy until its evidence is preserved.

Cut Over Without Creating Two Writers

Record the approved endpoint change, drain stale connection pools, update the secret or service-discovery record through the normal controlled mechanism, and start a canary application instance first. Confirm it resolves the new destination and that no worker still targets the old writer. Gradually restore read traffic, then writes, then asynchronous consumers while watching errors, latency, locks, replication, and business canaries. Keep the old instance fenced and retained according to the evidence policy.

A simple switch back is unsafe after new writes occur because the two databases now have different histories. Rollback requires a predeclared reconciliation plan: fence again, preserve both sides, identify writes accepted after cutover, and let the data owner choose replay, merge, or a new recovery point. Never improvise bidirectional synchronization during the incident.

Close the Incident With Measured Results

Record the actual RTO, the newest verified recovered transaction, estimated and confirmed data loss, every manual step, validation output, configuration drift, and time spent waiting on authorization. Restore redundancy and backup coverage before closing. Run the exercise on a schedule derived from business risk and after material changes to engine versions, topology, networking, secrets, or deployment tooling. A drill passes only when recovery evidence meets the objective; completing the provider operation is not enough. For planning backup frequency and recovery tiers, use the database backup strategy guide.

Official Primary Documentation

• Amazon RDS: restoring a DB instance to a specified time
• Amazon RDS: snapshot restore behavior and lazy loading
• AWS Well-Architected: plan for disaster recovery
• PostgreSQL: backup and restore methods