Database Reliability Engineering
Database SRE: Applying Reliability Engineering Principles Specifically to Your Database Tier
Database Site Reliability Engineering (DBRE) applies software engineering discipline, multi-window error budgets, and chaos engineering specifically to the stateful database tier. JusDB delivers dedicated 24/7 DBRE coverage, RTO/RPO durability engineering, automated failover verification, and toil reduction under guaranteed 99.999% availability and contractual sub-15-minute P1 incident response SLAs.
Generic SRE covers the whole stack. Database SRE goes deep on the layer that causes most P1 incidents — the database. JusDB defines database-specific SLOs, engineers for your RTO/RPO targets, runs chaos experiments against your database topology, and writes DB-specific runbooks your on-call engineers actually use.
Building an organisation-wide SRE practice with team structure, Kubernetes, and on-call tooling? See our Enterprise SRE Consulting service →
Signals That Matter
Database SLOs: What to Measure and Why
Generic SLOs measure HTTP success rate and p99 latency. Database SLOs must measure deeper — replication lag, query latency at percentile, connection pool saturation, deadlock rate, and replication divergence. JusDB defines the right SLOs for your database technology and workload.
MySQL / MariaDB SLOs
- Replication lag SLO: < 5s (alert), < 30s (breach)
- Query p99 latency SLO: < 100ms for OLTP
- Connection pool utilisation SLO: < 80%
- Deadlock rate SLO: < 10/hour
- Slow query rate SLO: < 0.1% of total queries
PostgreSQL SLOs
- Replication lag SLO: < 10s for standbys
- WAL sender/receiver lag SLO: < 100MB
- Vacuum bloat SLO: table bloat < 20%
- Lock wait time SLO: < 5s average
- Connection utilisation SLO: < 85% of max_connections
MongoDB SLOs
- Replication oplog lag SLO: < 10s
- Operation latency SLO: reads p99 < 50ms
- Working set in RAM SLO: > 90% cache hit
- Connection pool saturation SLO: < 75%
- Replica set election frequency SLO: < 1/week
Cassandra SLOs
- Read latency p99 SLO: < 10ms
- Write latency p99 SLO: < 5ms
- Compaction queue depth SLO: < 20 pending
- Dropped messages SLO: 0 dropped/hour
- Hinted handoff pending SLO: < 1,000
Rehearse The Failure
Database Chaos Engineering
You cannot know your database will survive a failure until you test it. JusDB runs controlled chaos experiments against your database topology — in staging first, then production — to verify your HA mechanisms work as expected.
Primary Crash Test
Kill the primary database process. Measure: time to failover, whether application reconnects, whether any writes are lost, whether the secondary correctly promotes.
Network Partition
Inject network latency and packet loss between primary and replica. Verify: split-brain protection activates, replication lag alerts fire, application degrades gracefully.
DCS Failure (Patroni/etcd)
Take etcd or Consul offline. Verify: Patroni enters read-only mode, no split-brain occurs, DCS recovery restores cluster to normal automatically.
Slow Query Storm
Inject a workload of slow queries to simulate a bad deployment. Verify: slow query alerts fire, query killer activates, connection pool degrades gracefully rather than exhausting all connections.
Disk Fill Simulation
Fill database disk to 80%, then 95%, then 100%. Verify: alerts fire at each threshold, database pauses writes at 95% before crash at 100%, recovery procedure works.
Backup & Recovery Drill
Actually restore from backup in a staging environment. Measure recovery time. Verify backup integrity. Most teams discover their backup restoration procedure is broken only during this exercise.
Recovery Math
RTO & RPO Engineering
RTO (Recovery Time Objective) and RPO (Recovery Point Objective) are promises. JusDB engineers the actual mechanisms that make those promises achievable — and tests them to verify they hold.
RTO Engineering
Time from failure detection to database serving traffic again.
RPO Engineering
Maximum data loss acceptable in a failure scenario.
Information Gain · High-Consequence Edge Cases
Database Reliability: Critical Production Failure Modes
Stateful database infrastructure exhibits complex distributed failure modes. Here are the 3 critical production failure patterns our dedicated DBRE team permanently mitigates:
Split-Brain Distributed Consensus Failure During Network Partitions
During cross-AZ network degradation or transient switch partitions, automated failover agents (such as uncalibrated Patroni/etcd or MHA clusters) may simultaneously elect two primaries. Asynchronous writes diverge across split nodes, corrupting transactional consistency and requiring manual hex-level binlog/WAL forensics to reconcile.
Enforce strict quorum consensus (odd-numbered Raft/DCS clusters), configure hardware-assisted watchdog fencing (STONITH/wdctl), deploy synchronous replication with remote_apply where RPO=0 is required, and run automated partition chaos tests.
Connection Thundering Herd & Thread Pool Starvation Cascade
When upstream microservices deploy or restart simultaneously, hundreds of application pods flood the database with unpooled client connections. Post-restart connection stampedes saturate max_connections, consume all available memory for backend processes, and lock operating system thread schedulers into CPU thrashing.
Architect multi-tier connection pooling (PgBouncer in transaction mode or ProxySQL with multiplexing), enforce client TCP backoff jitter, implement server-side connection queuing, and alert on connection pool utilization exceeding 75%.
Autovacuum / Compaction Starvation & Transaction ID Wraparound
Heavy write-throughput OLTP workloads generate millions of dead tuples or SSTables faster than background workers can clean them. Blocked autovacuum (caused by long-running transactions or advisory locks) drives table bloat, degrades buffer cache hit ratios, and risks catastrophic PostgreSQL emergency freeze shutdown (wraparound failure).
Tune cost-based vacuum limits (autovacuum_vacuum_cost_limit = 2000), enable multi-worker vacuum parallelism, deploy automated alerts for oldest transaction age exceeding 500 million transactions, and kill orphaned idle-in-transaction sessions automatically.
Our DBREs execute non-blocking Prometheus burn-rate queries and lock-graph queries to identify degraded SLIs and blocking transactions without acquiring table locks:
# 1-hour and 6-hour burn rate for query latency SLO (<100ms)
(
sum(rate(database_query_duration_seconds_bucket{le="0.1"}[1h]))
/
sum(rate(database_query_duration_seconds_count[1h]))
) < 0.999
and
(
sum(rate(database_query_duration_seconds_bucket{le="0.1"}[6h]))
/
sum(rate(database_query_duration_seconds_count[6h]))
) < 0.999# Inspect blocked lock chains without table acquisition
SELECT blocked.pid AS blocked_pid,
blocking.pid AS blocking_pid,
act.usename AS blocked_user,
now() - act.query_start AS duration,
act.query AS blocked_statement
FROM pg_catalog.pg_locks blocked
JOIN pg_catalog.pg_stat_activity act ON act.pid = blocked.pid
JOIN pg_catalog.pg_locks blocking
ON blocking.locktype = blocked.locktype
AND blocking.database IS NOT DISTINCT FROM blocked.database
AND blocking.relation IS NOT DISTINCT FROM blocked.relation
AND blocking.pid != blocked.pid
WHERE NOT blocked.granted;Comparative Architecture Matrix · Database Reliability
Dedicated DBRE vs Generic SRE vs Reactive DBA
Compare JusDB dedicated Database Reliability Engineering against generic cloud platform SRE teams and traditional ticket-driven DBAs across error budget engineering, chaos drills, toil elimination, and 24/7 SLAs.
| Reliability Vector | JusDB Dedicated DBRE | Generic Cloud SRE | Traditional Reactive DBA |
|---|---|---|---|
| SLO, SLI & Multi-Window Error Budget Engineering | Engine-specific database SLIs (p99 query latency, replication lag, connection saturation, lock contention) with multi-window burn-rate alerting. | High-level HTTP availability and service-level latency SLOs that completely miss deep database storage engine stalls and transactional lock deadlocks. | No SLOs or error budgets; reactive management driven by arbitrary static alerts (CPU > 80%, disk > 85%) causing severe alert fatigue. |
| High Availability & Chaos Engineering Drills | GameDay chaos experiments (split-brain injection, primary crash, network partitioning, DCS outage) validating automated sub-30s failover without data loss. | Pod/node restart testing in Kubernetes; does not test stateful quorum consensus, synchronous replication durability, or application pool reconnection. | Zero chaos testing; failover mechanisms remain untested until an unpredicted production crash occurs, resulting in multi-hour downtime. |
| RTO / RPO Durability & PITR Verification | Formally engineered RTO (<30s) and RPO (=0 or <5s) architectures with automated daily snapshot and continuous WAL restore drills to staging environments. | Relies on cloud-native automated backups without verifying restore times, data consistency, or database engine startup integrity. | Manual daily pg_dump / mysqldump scripts stored on shared disks; recovery drills rarely executed until emergency disaster recovery is required. |
| Toil Elimination & Infrastructure as Code (IaC) | Strict 50% toil cap rule; repetitive provisioning, schema migrations, and failover runbooks codified into Terraform, Ansible, and Kubernetes operators. | Focuses IaC on stateless microservices and ingress controllers, treating databases as black-box managed services with default configurations. | 80–90% operational toil; manual parameter changes via GUI or SSH, ad-hoc shell scripts, and ticket-driven repetitive administration. |
| Observability, Telemetry & Root Cause Forensics | Deep database engine instrumentation (pg_stat_statements, Performance Schema, slowlog, wait-event sampling) integrated with Grafana and OpenTelemetry. | Infrastructure-level APM metrics (host CPU, memory, network I/O) that fail to explain internal engine buffer bloat or lock graph cycles. | Ad-hoc query execution during active incidents (top, ps, SHOW FULL PROCESSLIST); historical query execution telemetry is missing. |
| Incident Response, Blameless Postmortems & 24/7 SLA | Guaranteed <15m P1 incident response with senior DBREs joining the war room; structured 48-hour blameless postmortems driving permanent architectural fixes. | Generalist on-call rotation with limited database engine internals depth; escalates database incidents to external vendors or developers. | Best-effort on-call with 1–4 hour response times; postmortems focus on human blame rather than systemic resilience improvements. |
Questions
FAQ
Make your databases reliably reliable
JusDB defines your database SLOs, tests your HA mechanisms, writes your runbooks, and takes your on-call for the database tier.