Database Reliability Engineering
Database SRE: Applying Reliability Engineering Principles Specifically to Your Database Tier
Database Site Reliability Engineering (DBRE) applies software engineering discipline, multi-window error budgets, and chaos engineering specifically to the stateful database tier. JusDB delivers dedicated DBRE coverage, RTO/RPO durability engineering, automated failover verification, and toil reduction. Availability and response commitments are defined in your service agreement. Initial-response targets are 2 to 4 hours during covered business hours, Mon–Fri 09:00–18:00 IST (UTC+5:30). Where your service agreement includes 24x7 on-call, the initial-response target for P1 incidents is 30 minutes. Targets are not resolution guarantees.
Generic SRE covers the whole stack. Database SRE goes deep on the layer that causes most P1 incidents — the database. JusDB defines database-specific SLOs, engineers for your RTO/RPO targets, runs chaos experiments against your database topology, and writes DB-specific runbooks your on-call engineers actually use.
Building an organisation-wide SRE practice with team structure, Kubernetes, and on-call tooling? See our Enterprise SRE Consulting service →
Database SLOs: What to Measure and Why
Generic SLOs measure HTTP success rate and p99 latency. Database SLOs must measure deeper — replication lag, query latency at percentile, connection pool saturation, deadlock rate, and replication divergence. JusDB defines the right SLOs for your database technology and workload.
MySQL / MariaDB SLOs
- Replication lag SLO: < 5s (alert), < 30s (breach)
- Query p99 latency SLO: < 100ms for OLTP
- Connection pool utilisation SLO: < 80%
- Deadlock rate SLO: < 10/hour
- Slow query rate SLO: < 0.1% of total queries
PostgreSQL SLOs
- Replication lag SLO: < 10s for standbys
- WAL sender/receiver lag SLO: < 100MB
- Vacuum bloat SLO: table bloat < 20%
- Lock wait time SLO: < 5s average
- Connection utilisation SLO: < 85% of max_connections
MongoDB SLOs
- Replication oplog lag SLO: < 10s
- Operation latency SLO: reads p99 < 50ms
- Working set in RAM SLO: > 90% cache hit
- Connection pool saturation SLO: < 75%
- Replica set election frequency SLO: < 1/week
Cassandra SLOs
- Read latency p99 SLO: < 10ms
- Write latency p99 SLO: < 5ms
- Compaction queue depth SLO: < 20 pending
- Dropped messages SLO: 0 dropped/hour
- Hinted handoff pending SLO: < 1,000
Database Chaos Engineering
You cannot know your database will survive a failure until you test it. JusDB runs controlled chaos experiments against your database topology — in staging first, then production — to verify your HA mechanisms work as expected.
Primary Crash Test
Kill the primary database process. Measure: time to failover, whether application reconnects, whether any writes are lost, whether the secondary correctly promotes.
Network Partition
Inject network latency and packet loss between primary and replica. Verify: split-brain protection activates, replication lag alerts fire, application degrades gracefully.
DCS Failure (Patroni/etcd)
Take etcd or Consul offline. Verify: Patroni enters read-only mode, no split-brain occurs, DCS recovery restores cluster to normal automatically.
Slow Query Storm
Inject a workload of slow queries to simulate a bad deployment. Verify: slow query alerts fire, query killer activates, connection pool degrades gracefully rather than exhausting all connections.
Disk Fill Simulation
Fill database disk to 80%, then 95%, then 100%. Verify: alerts fire at each threshold, database pauses writes at 95% before crash at 100%, recovery procedure works.
Backup & Recovery Drill
Actually restore from backup in a staging environment. Measure recovery time. Verify backup integrity. Most teams discover their backup restoration procedure is broken only during this exercise.
RTO & RPO Engineering
RTO (Recovery Time Objective) and RPO (Recovery Point Objective) are promises. JusDB engineers the actual mechanisms that make those promises achievable — and tests them to verify they hold.
RTO Engineering
Time from failure detection to database serving traffic again.
RPO Engineering
Maximum data loss acceptable in a failure scenario.
Database Reliability: Critical Production Failure Modes
Stateful database infrastructure exhibits complex distributed failure modes. Here are the 3 critical production failure patterns our dedicated DBRE team permanently mitigates:
Split-Brain Distributed Consensus Failure During Network Partitions
During cross-AZ network degradation or transient switch partitions, automated failover agents (such as uncalibrated Patroni/etcd or MHA clusters) may simultaneously elect two primaries. Asynchronous writes diverge across split nodes, corrupting transactional consistency and requiring manual hex-level binlog/WAL forensics to reconcile.
Enforce strict quorum consensus (odd-numbered Raft/DCS clusters), configure hardware-assisted watchdog fencing (STONITH/wdctl), deploy synchronous replication with remote_apply where RPO=0 is required, and run automated partition chaos tests.
Connection Thundering Herd & Thread Pool Starvation Cascade
When upstream microservices deploy or restart simultaneously, hundreds of application pods flood the database with unpooled client connections. Post-restart connection stampedes saturate max_connections, consume all available memory for backend processes, and lock operating system thread schedulers into CPU thrashing.
Architect multi-tier connection pooling (PgBouncer in transaction mode or ProxySQL with multiplexing), enforce client TCP backoff jitter, implement server-side connection queuing, and alert on connection pool utilization exceeding 75%.
Autovacuum / Compaction Starvation & Transaction ID Wraparound
Heavy write-throughput OLTP workloads generate millions of dead tuples or SSTables faster than background workers can clean them. Autovacuum held back by long-running transactions or stale replication slots drives table bloat and lowers buffer cache hit ratios; left unchecked, XID age climbs toward the wraparound limit, where PostgreSQL stops assigning transaction IDs and writes fail.
Tune cost-based vacuum limits (autovacuum_vacuum_cost_limit = 2000), enable multi-worker vacuum parallelism, deploy automated alerts for oldest transaction age exceeding 500 million transactions, and end orphaned idle-in-transaction sessions automatically with idle_in_transaction_session_timeout.
Our DBREs execute non-blocking Prometheus burn-rate queries and lock-graph queries to identify degraded SLIs and blocking transactions without acquiring table locks:
# 1-hour and 6-hour burn rate for query latency SLO (<100ms)
(
sum(rate(database_query_duration_seconds_bucket{le="0.1"}[1h]))
/
sum(rate(database_query_duration_seconds_count[1h]))
) < 0.999
and
(
sum(rate(database_query_duration_seconds_bucket{le="0.1"}[6h]))
/
sum(rate(database_query_duration_seconds_count[6h]))
) < 0.999# Inspect blocked lock chains without table acquisition
SELECT blocked.pid AS blocked_pid,
blocking.pid AS blocking_pid,
act.usename AS blocked_user,
now() - act.query_start AS duration,
act.query AS blocked_statement
FROM pg_catalog.pg_locks blocked
JOIN pg_catalog.pg_stat_activity act ON act.pid = blocked.pid
JOIN pg_catalog.pg_locks blocking
ON blocking.locktype = blocked.locktype
AND blocking.database IS NOT DISTINCT FROM blocked.database
AND blocking.relation IS NOT DISTINCT FROM blocked.relation
AND blocking.pid != blocked.pid
WHERE NOT blocked.granted;Dedicated DBRE vs Generic SRE vs Reactive DBA
Compare JusDB dedicated Database Reliability Engineering against generic cloud platform SRE teams and traditional ticket-driven DBAs across error budget engineering, chaos drills, toil elimination, and response coverage.
| Reliability Vector | JusDB Dedicated DBRE | Generic Cloud SRE | Traditional Reactive DBA |
|---|---|---|---|
| SLO, SLI & Multi-Window Error Budget Engineering | Engine-specific database SLIs (p99 query latency, replication lag, connection saturation, lock contention) with multi-window burn-rate alerting. | High-level HTTP availability and service-level latency SLOs that completely miss deep database storage engine stalls and transactional lock deadlocks. | No SLOs or error budgets; reactive management driven by arbitrary static alerts (CPU > 80%, disk > 85%) causing severe alert fatigue. |
| High Availability & Chaos Engineering Drills | GameDay chaos experiments where the engagement includes them (split-brain injection, primary crash, network partitioning, DCS outage), validating automated failover against the RTO and RPO targets set for each workload. | Pod/node restart testing in Kubernetes; does not test stateful quorum consensus, synchronous replication durability, or application pool reconnection. | Zero chaos testing; failover mechanisms remain untested until an unpredicted production crash occurs, resulting in multi-hour downtime. |
| RTO / RPO Durability & PITR Verification | RTO and RPO architectures engineered to the targets set for each workload (RPO = 0 where synchronous replication is in place), with automated snapshot and WAL restore drills to staging on the cadence agreed in your proposal. | Relies on cloud-native automated backups without verifying restore times, data consistency, or database engine startup integrity. | Manual daily pg_dump / mysqldump scripts stored on shared disks; recovery drills rarely executed until emergency disaster recovery is required. |
| Toil Elimination & Infrastructure as Code (IaC) | Strict 50% toil cap rule; repetitive provisioning, schema migrations, and failover runbooks codified into Terraform, Ansible, and Kubernetes operators. | Focuses IaC on stateless microservices and ingress controllers, treating databases as black-box managed services with default configurations. | Mostly operational toil: manual parameter changes via GUI or SSH, ad-hoc shell scripts, and ticket-driven repetitive administration. |
| Observability, Telemetry & Root Cause Forensics | Deep database engine instrumentation (pg_stat_statements, Performance Schema, slowlog, wait-event sampling) integrated with Grafana and OpenTelemetry. | Infrastructure-level APM metrics (host CPU, memory, network I/O) that fail to explain internal engine buffer bloat or lock graph cycles. | Ad-hoc query execution during active incidents (top, ps, SHOW FULL PROCESSLIST); historical query execution telemetry is missing. |
| Incident Response, On-Call Coverage & Blameless Postmortems | Agreed initial-response targets (24x7 on-call available) with senior DBREs joining the war room; blameless postmortems after incidents that consume significant error budget, on the timeline in your service agreement, driving architectural fixes. | Generalist on-call rotation with limited database engine internals depth; escalates database incidents to external vendors or developers. | Best-effort on-call with 1–4 hour response times; postmortems focus on human blame rather than systemic resilience improvements. |
FAQ
Make your databases reliably reliable
JusDB defines your database SLOs, tests your HA mechanisms, writes your runbooks, and takes your on-call for the database tier.