| Sentinel Quorum & Multi-AZ Monitor Placement | 3 or 5-node odd-quorum Sentinel mesh distributed across independent availability zones with strict anti-affinity rules, co-located application monitoring, and zero single-point-of-failure topologies. | Even-numbered Sentinel nodes deployed on same host or single AZ; network partition induces 50/50 vote split and total election deadlock. | Single Sentinel instance monitoring cluster; any Sentinel crash or transient partition completely disables automated cluster failover. | Sentinels deployed on same compute instances as Valkey nodes; node crash simultaneously eliminates monitor and quorum. |
| Asynchronous Replication Lag & Buffer Sizing | Sub-second lag tracking via master_link_status and replication-offset deltas, paired with calibrated repl-backlog-size and client-output-buffer-limit tuning to prevent full resync loops. | Default replication backlog buffers triggering full resync disk thrashing whenever network jitter exceeds 50ms. | Ignores replication lag metrics until lagging replicas get promoted with massive data staleness during unexpected primary failures. | Unmonitored async replication causing silent write divergence and unbounded memory growth under sustained load. |
| Split-Brain Prevention (min-replicas-to-write) | Enforced min-replicas-to-write and min-replicas-max-lag constraints on primary nodes, isolating partitioned primaries and blocking phantom writes during network splits. | Unconfigured min-replicas-to-write; partitioned primary continues accepting isolated writes that are permanently wiped upon healing. | Misunderstands partition semantics; relies on client timeouts rather than engine-level fencing to prevent dual-primary states. | Completely disables replica write guards, causing catastrophic data corruption when both partitions accept concurrent conflicting keys. |
| Automated Primary Failover & Client Discovery | Calibrated down-after-milliseconds, fast Sentinel election, parallel replica resync, and client pub/sub +switch-master reconnection protocols, with failover timing tested against your RTO target. | Default 30s failure detection combined with unhandled client connection caching resulting in 60s+ application write outages. | Requires manual administrative intervention and manual DNS updates to redirect traffic following primary host failure. | Hardcoded IP endpoints in client configuration; failover leaves application persistently attempting writes to dead node. |
| Cross-Region Disaster Recovery & RPO Targets | Engineered cross-region asynchronous replication standby, automated point-in-time RDB/AOF archival to object storage, and an RPO target agreed per workload. | Daily snapshot backups stored locally without cross-region replication, resulting in 24-hour RPO data loss during regional outages. | Manual off-site tar exports with unverified restore procedures; RTO exceeds 12 hours during catastrophic site failure. | No off-site backup or cross-region DR plan; single cloud region failure results in total, irreversible data loss. |
| Chaos Engineering & Sentinel Leader Election Drills | Automated chaos drills on an agreed cadence, simulating network partitions, hard node kills, and split-brain scenarios to validate quorum election and failover timing against your targets. | Untested failover assumptions; first real failover test occurs during an unannounced 3 AM production crisis. | Theoretical runbooks on paper without empirical failure testing; election timeouts fail under production connection stress. | Zero chaos testing or disaster recovery drills; panic-driven ad-hoc reconfiguration during infrastructure incidents. |