Free audit

View Audit Scope

Database Reliability Engineering

Database SRE: Applying Reliability Engineering Principles Specifically to Your Database Tier

Executive Direct Answer · Database SRE Heuristic

Database Site Reliability Engineering (DBRE) applies software engineering discipline, multi-window error budgets, and chaos engineering specifically to the stateful database tier. JusDB delivers dedicated 24/7 DBRE coverage, RTO/RPO durability engineering, automated failover verification, and toil reduction under guaranteed 99.999% availability and contractual sub-15-minute P1 incident response SLAs.

Response SLA: Contractual <15m P1 War Room·Availability: 99.999% Uptime Engineering·Recovery: RTO < 30s · RPO = 0 / < 5s·Mandate: <50% Toil Cap Rule·Compliance: ISO 27001 & SOC 2

Generic SRE covers the whole stack. Database SRE goes deep on the layer that causes most P1 incidents — the database. JusDB defines database-specific SLOs, engineers for your RTO/RPO targets, runs chaos experiments against your database topology, and writes DB-specific runbooks your on-call engineers actually use.

Building an organisation-wide SRE practice with team structure, Kubernetes, and on-call tooling? See our Enterprise SRE Consulting service →

Signals That Matter

Database SLOs: What to Measure and Why

Generic SLOs measure HTTP success rate and p99 latency. Database SLOs must measure deeper — replication lag, query latency at percentile, connection pool saturation, deadlock rate, and replication divergence. JusDB defines the right SLOs for your database technology and workload.

MySQL / MariaDB SLOs

  • Replication lag SLO: < 5s (alert), < 30s (breach)
  • Query p99 latency SLO: < 100ms for OLTP
  • Connection pool utilisation SLO: < 80%
  • Deadlock rate SLO: < 10/hour
  • Slow query rate SLO: < 0.1% of total queries

PostgreSQL SLOs

  • Replication lag SLO: < 10s for standbys
  • WAL sender/receiver lag SLO: < 100MB
  • Vacuum bloat SLO: table bloat < 20%
  • Lock wait time SLO: < 5s average
  • Connection utilisation SLO: < 85% of max_connections

MongoDB SLOs

  • Replication oplog lag SLO: < 10s
  • Operation latency SLO: reads p99 < 50ms
  • Working set in RAM SLO: > 90% cache hit
  • Connection pool saturation SLO: < 75%
  • Replica set election frequency SLO: < 1/week

Cassandra SLOs

  • Read latency p99 SLO: < 10ms
  • Write latency p99 SLO: < 5ms
  • Compaction queue depth SLO: < 20 pending
  • Dropped messages SLO: 0 dropped/hour
  • Hinted handoff pending SLO: < 1,000

Rehearse The Failure

Database Chaos Engineering

You cannot know your database will survive a failure until you test it. JusDB runs controlled chaos experiments against your database topology — in staging first, then production — to verify your HA mechanisms work as expected.

Primary Crash Test

Kill the primary database process. Measure: time to failover, whether application reconnects, whether any writes are lost, whether the secondary correctly promotes.

Network Partition

Inject network latency and packet loss between primary and replica. Verify: split-brain protection activates, replication lag alerts fire, application degrades gracefully.

DCS Failure (Patroni/etcd)

Take etcd or Consul offline. Verify: Patroni enters read-only mode, no split-brain occurs, DCS recovery restores cluster to normal automatically.

Slow Query Storm

Inject a workload of slow queries to simulate a bad deployment. Verify: slow query alerts fire, query killer activates, connection pool degrades gracefully rather than exhausting all connections.

Disk Fill Simulation

Fill database disk to 80%, then 95%, then 100%. Verify: alerts fire at each threshold, database pauses writes at 95% before crash at 100%, recovery procedure works.

Backup & Recovery Drill

Actually restore from backup in a staging environment. Measure recovery time. Verify backup integrity. Most teams discover their backup restoration procedure is broken only during this exercise.

Recovery Math

RTO & RPO Engineering

RTO (Recovery Time Objective) and RPO (Recovery Point Objective) are promises. JusDB engineers the actual mechanisms that make those promises achievable — and tests them to verify they hold.

RTO Engineering

Time from failure detection to database serving traffic again.

RTO < 60sPatroni/repmgrd automatic failover with pre-warmed standby
RTO < 5 minAutomated runbook: promote standby, update DNS/VIP, restart application connections
RTO < 30 minManual failover with documented step-by-step runbook, pg_basebackup from recent snapshot
RTO < 4 hoursFull restore from PITR backup to new instance

RPO Engineering

Maximum data loss acceptable in a failure scenario.

RPO = 0Synchronous replication to standby (PostgreSQL synchronous_commit=remote_apply)
RPO < 5sAsynchronous streaming replication with monitored replication lag SLO
RPO < 1 minWAL archiving to S3 every 60 seconds (pgBackRest / Barman)
RPO < 24hDaily automated snapshot with integrity verification

Information Gain · High-Consequence Edge Cases

Database Reliability: Critical Production Failure Modes

Stateful database infrastructure exhibits complex distributed failure modes. Here are the 3 critical production failure patterns our dedicated DBRE team permanently mitigates:

CRITICAL P1

Split-Brain Distributed Consensus Failure During Network Partitions

During cross-AZ network degradation or transient switch partitions, automated failover agents (such as uncalibrated Patroni/etcd or MHA clusters) may simultaneously elect two primaries. Asynchronous writes diverge across split nodes, corrupting transactional consistency and requiring manual hex-level binlog/WAL forensics to reconcile.

JusDB Engineering Mitigation:

Enforce strict quorum consensus (odd-numbered Raft/DCS clusters), configure hardware-assisted watchdog fencing (STONITH/wdctl), deploy synchronous replication with remote_apply where RPO=0 is required, and run automated partition chaos tests.

HIGH SEV-2

Connection Thundering Herd & Thread Pool Starvation Cascade

When upstream microservices deploy or restart simultaneously, hundreds of application pods flood the database with unpooled client connections. Post-restart connection stampedes saturate max_connections, consume all available memory for backend processes, and lock operating system thread schedulers into CPU thrashing.

JusDB Engineering Mitigation:

Architect multi-tier connection pooling (PgBouncer in transaction mode or ProxySQL with multiplexing), enforce client TCP backoff jitter, implement server-side connection queuing, and alert on connection pool utilization exceeding 75%.

HIGH SEV-2

Autovacuum / Compaction Starvation & Transaction ID Wraparound

Heavy write-throughput OLTP workloads generate millions of dead tuples or SSTables faster than background workers can clean them. Blocked autovacuum (caused by long-running transactions or advisory locks) drives table bloat, degrades buffer cache hit ratios, and risks catastrophic PostgreSQL emergency freeze shutdown (wraparound failure).

JusDB Engineering Mitigation:

Tune cost-based vacuum limits (autovacuum_vacuum_cost_limit = 2000), enable multi-worker vacuum parallelism, deploy automated alerts for oldest transaction age exceeding 500 million transactions, and kill orphaned idle-in-transaction sessions automatically.

Telemetry Runbooks · Non-Blocking DBRE Diagnostic Forensics

Our DBREs execute non-blocking Prometheus burn-rate queries and lock-graph queries to identify degraded SLIs and blocking transactions without acquiring table locks:

Multi-Window Error Budget Burn Rate QueryPromQL
# 1-hour and 6-hour burn rate for query latency SLO (<100ms)
(
  sum(rate(database_query_duration_seconds_bucket{le="0.1"}[1h]))
  /
  sum(rate(database_query_duration_seconds_count[1h]))
) < 0.999
and
(
  sum(rate(database_query_duration_seconds_bucket{le="0.1"}[6h]))
  /
  sum(rate(database_query_duration_seconds_count[6h]))
) < 0.999
Lock Graph & Transaction Blocking ForensicsSQL / CLI
# Inspect blocked lock chains without table acquisition
SELECT blocked.pid AS blocked_pid,
       blocking.pid AS blocking_pid,
       act.usename AS blocked_user,
       now() - act.query_start AS duration,
       act.query AS blocked_statement
FROM pg_catalog.pg_locks blocked
JOIN pg_catalog.pg_stat_activity act ON act.pid = blocked.pid
JOIN pg_catalog.pg_locks blocking 
    ON blocking.locktype = blocked.locktype
    AND blocking.database IS NOT DISTINCT FROM blocked.database
    AND blocking.relation IS NOT DISTINCT FROM blocked.relation
    AND blocking.pid != blocked.pid
WHERE NOT blocked.granted;

Comparative Architecture Matrix · Database Reliability

Dedicated DBRE vs Generic SRE vs Reactive DBA

Compare JusDB dedicated Database Reliability Engineering against generic cloud platform SRE teams and traditional ticket-driven DBAs across error budget engineering, chaos drills, toil elimination, and 24/7 SLAs.

Reliability VectorJusDB Dedicated DBREGeneric Cloud SRETraditional Reactive DBA
SLO, SLI & Multi-Window Error Budget EngineeringEngine-specific database SLIs (p99 query latency, replication lag, connection saturation, lock contention) with multi-window burn-rate alerting.High-level HTTP availability and service-level latency SLOs that completely miss deep database storage engine stalls and transactional lock deadlocks.No SLOs or error budgets; reactive management driven by arbitrary static alerts (CPU > 80%, disk > 85%) causing severe alert fatigue.
High Availability & Chaos Engineering DrillsGameDay chaos experiments (split-brain injection, primary crash, network partitioning, DCS outage) validating automated sub-30s failover without data loss.Pod/node restart testing in Kubernetes; does not test stateful quorum consensus, synchronous replication durability, or application pool reconnection.Zero chaos testing; failover mechanisms remain untested until an unpredicted production crash occurs, resulting in multi-hour downtime.
RTO / RPO Durability & PITR VerificationFormally engineered RTO (<30s) and RPO (=0 or <5s) architectures with automated daily snapshot and continuous WAL restore drills to staging environments.Relies on cloud-native automated backups without verifying restore times, data consistency, or database engine startup integrity.Manual daily pg_dump / mysqldump scripts stored on shared disks; recovery drills rarely executed until emergency disaster recovery is required.
Toil Elimination & Infrastructure as Code (IaC)Strict 50% toil cap rule; repetitive provisioning, schema migrations, and failover runbooks codified into Terraform, Ansible, and Kubernetes operators.Focuses IaC on stateless microservices and ingress controllers, treating databases as black-box managed services with default configurations.80–90% operational toil; manual parameter changes via GUI or SSH, ad-hoc shell scripts, and ticket-driven repetitive administration.
Observability, Telemetry & Root Cause ForensicsDeep database engine instrumentation (pg_stat_statements, Performance Schema, slowlog, wait-event sampling) integrated with Grafana and OpenTelemetry.Infrastructure-level APM metrics (host CPU, memory, network I/O) that fail to explain internal engine buffer bloat or lock graph cycles.Ad-hoc query execution during active incidents (top, ps, SHOW FULL PROCESSLIST); historical query execution telemetry is missing.
Incident Response, Blameless Postmortems & 24/7 SLAGuaranteed <15m P1 incident response with senior DBREs joining the war room; structured 48-hour blameless postmortems driving permanent architectural fixes.Generalist on-call rotation with limited database engine internals depth; escalates database incidents to external vendors or developers.Best-effort on-call with 1–4 hour response times; postmortems focus on human blame rather than systemic resilience improvements.

Questions

FAQ

Make your databases reliably reliable

JusDB defines your database SLOs, tests your HA mechanisms, writes your runbooks, and takes your on-call for the database tier.