Free audit

View Audit Scope
Valkey 8.0 HASentinel Quorum · Split-Brain Fencing · Dual-Channel Sync

Valkey High Availability & Sentinel Engineering

JusDB architects, hardens, and tests production Valkey 8.0 high availability across 3-node and 5-node Valkey Sentinel topologies, 16,384-slot Valkey Clusters, and Multi-AZ cloud deployments. Our engineers calibrate down-after-milliseconds and failover-timeout quorums, enforce min-replicas-to-write split-brain fencing, configure Valkey 8.0 dual-channel replication, and run scheduled chaos failover drills.

The first 30-minute call covers your databases, current ownership, and required coverage. No production access is needed. Or request a free audit of one database instance.

  • NDA before access
  • Read-only by default
  • Written scope before work starts

Valkey HA & Sentinel quorum controls

Select an HA subsystem to inspect

3-Node & 5-Node Odd Quorum Placement

Quorum 2 / 3 across 3 AZs

Valkey Sentinel requires an odd majority of Sentinel processes across independent failure domains to transition a primary from Subjective Down (+sdown) to Objective Down (+odown) and elect a Raft leader.

Deploying Sentinels on only 2 availability zones or co-locating all Sentinels with a single primary creates failover paralysis if that AZ isolates.

Inspect hardened sentinel.conf & fencing rules

Verified against Linux Foundation Valkey 8.0 Sentinel and replication specifications.

HA topologies engineered
3-node and 5-node Valkey Sentinel, 16,384-slot Valkey Cluster, HAProxy TCP routing, and Multi-AZ ElastiCache / MemoryDB
Initial-response targets
15 minutes for a critical (P1) incident under a 24x7 support agreement; 2 to 4 hours in business hours. See details
Coverage hours
Business hours (03:30–12:30 UTC) by default; 24x7 on-call available under a 24x7 support agreement

Selected clients

  • Innoviti
  • Blitz
  • Freecharge
  • Nango
  • Recochain
  • Screen Facts
  • Nexora
  • WeCredit

Production failure modes

Common Valkey failover and replication failures we resolve

Simply running valkey-sentinel processes does not guarantee high availability; misconfigured quorums, unfenced primaries, and uncalibrated client pools turn transient network blips into prolonged outages.

Valkey high availability symptoms, root causes, and first diagnostic actions
Sentinel / Valkey signalWhat is usually happeningWhere we start
+odown Split-BrainBoth the old primary and promoted replica accept writes during an AZ partitionNetwork partition isolates AZ-a containing the primary; Sentinels in AZ-b and AZ-c promote a replica, while clients inside AZ-a keep writing to the isolated primary until the link heals and those writes are wiped.Configure min-replicas-to-write 1 and min-replicas-max-lag 5 on all data nodes so an isolated primary rejects writes within 5 seconds of losing replica heartbeats.
-failover-abort-not-electedSentinel detects +sdown on primary but never promotes a replicaOnly 2 Sentinel processes were deployed (or 2 of 3 Sentinels lived in the failed AZ), leaving the remaining Sentinel unable to win the majority vote (> N/2 + 1) required to authorize failover.Deploy 3 or 5 Sentinel daemons distributed strictly across 3 Availability Zones, verify SENTINEL CKQUORUM, and synchronize system clocks via Chrony/NTP.
False +sdown FlappingSentinel triggers unnecessary failovers during BGSAVE or Lua spikesdown-after-milliseconds is set too aggressively (for example 1000ms); a brief copy-on-write fork stall or slow Lua script delays PING responses and triggers a false promotion cascade.Calibrate down-after-milliseconds to 5000ms and failover-timeout to 60000ms, disable Linux THP, and enable Valkey 8.0 io-threads so health checks stay responsive.
Replica Resync AvalanchePromoted primary runs out of memory immediately after failoverFollowing failover, multiple replicas fail partial PSYNC2 and simultaneously request a full RDB resync from the newly promoted primary, multiplying client-output-buffer-limit replica memory.Enable Valkey 8.0 dual-channel replication with repl-diskless-sync yes, raise repl-backlog-size to 1-2 GiB, and cap parallel-syncs 1 in sentinel.conf.
Stale Primary ReconnectMicroservices keep writing to the demoted primary (READONLY errors) after failoverApplication pools hardcode the primary IP/DNS instead of subscribing to Sentinel +switch-master events or routing through a health-checked HAProxy / Envoy TCP tier.Configure Sentinel-aware drivers (Lettuce, ioredis, go-redis), enforce CLIENT KILL type normal in Sentinel client-reconfig-script, or deploy HAProxy TCP health checks checking role:master.
Empty RDB Data WipeCrashed primary reboots with an empty disk and wipes all replicasA primary running with persistence disabled (save '') crashes, auto-restarts via systemd with 0 keys, and replicates an empty RDB to every attached replica before Sentinel intervenes.Never enable unattended systemd auto-restart on persistence-disabled primaries; require hybrid RDB+AOF persistence or start recovered nodes as replicas first.

Technical scope

What does our Valkey high availability service include?

We design, configure, and chaos-test your Valkey Sentinel or Cluster failover topology from kernel networking up to your application connection pools.

  • 3-AZ Sentinel quorum & Raft election design

    We architect odd-numbered 3-node and 5-node Valkey Sentinel topologies across independent Availability Zones so any single node or zone failure preserves failover majority.

    • Calibrate sentinel monitor quorum, down-after-milliseconds, parallel-syncs, and failover-timeout
    • Configure sentinel announce-ip / announce-hostnames for NAT, Docker, and Kubernetes networks

    SENTINEL CKQUORUM · down-after-milliseconds · failover-timeout · +switch-master

  • Split-brain fencing & write durability bounds

    Asynchronous replication cannot guarantee zero lost writes without explicit fencing. We bound your maximum data-loss window during network partitions.

    • Enforce min-replicas-to-write and min-replicas-max-lag across all primary candidates
    • Implement synchronous WAIT / WAITAOF semantics for critical financial or lock mutations

    min-replicas-to-write · min-replicas-max-lag · WAIT · WAITAOF

  • Valkey 8.0 dual-channel & diskless replication

    Post-failover recovery is when HA clusters are most fragile. We leverage Valkey 8.0 replication improvements to eliminate resync storms and disk bottlenecks.

    • Enable Valkey 8.0 dual-channel replication and repl-diskless-sync yes with tuned delay
    • Size repl-backlog-size and repl-backlog-ttl so transient blips resolve via PSYNC2

    dual-channel replication · repl-diskless-sync · PSYNC2 · repl-backlog-size

    Redis vs Valkey 8.0 replication
  • HAProxy, Envoy & Sentinel-aware client routing

    A 5-second Sentinel promotion is wasted if your application takes 3 minutes to discover the new primary. We harden client discovery and TCP load balancers.

    • Configure HAProxy TCP health checks probing INFO replication for role:master
    • Tune Lettuce, Jedis, ioredis, and go-redis Sentinel topology discovery and connection eviction

    HAProxy tcp-check · Envoy Redis proxy · +switch-master · CLIENT KILL

  • Sentinel vs 16,384-slot Cluster HA selection

    We help your team choose and transition between Sentinel master-replica HA and horizontally sharded Valkey Cluster with built-in gossip failover.

    • Model dataset growth, multi-key Lua requirements, and cross-AZ network transfer costs
    • Configure cluster-require-full-coverage and cluster-replica-validity-factor for sharded HA

    Valkey Sentinel · Valkey Cluster · cluster-node-timeout · replica-priority

    Valkey Cluster engineering
  • Chaos failover drills & RDB/AOF restore testing

    Untested high availability fails when paged at 03:00 UTC. We run controlled network partition and primary termination drills to prove your actual RTO and RPO.

    • Execute scheduled SENTINEL FAILOVER and iptables partition tests in staging and maintenance windows
    • Harden hybrid RDB + AOF (aof-use-rdb-preamble yes) backups with automated restore verification

    SENTINEL FAILOVER · DEBUG SLEEP · aof-use-rdb-preamble · S3 restore drills

    Valkey 24x7 support (where contracted)

Shared responsibility & response targets

Where default Sentinel setups stop and what JusDB engineers run

Default sentinel.conf templates leave split-brain fencing disabled and replication backlogs undersized; we engineer deterministic failover boundaries.

Comparison between default Valkey Sentinel setups and what JusDB database engineers configure
HA subsystemWhere default setups stopWhat JusDB engineers run
Sentinel quorum placementCo-locates 2 Sentinels on 2 Valkey hosts or places all 3 Sentinels inside a single Availability Zone or VM host group.Distributes 3 or 5 Sentinel processes across 3 independent AZs with verified SENTINEL CKQUORUM and anti-affinity rules.
Split-brain partition protectionLeaves min-replicas-to-write at 0, allowing an isolated primary to accept minutes of doomed writes during a network partition.Configures min-replicas-to-write 1 and min-replicas-max-lag 5, plus client-reconfig-script fencing to disconnect stale writers.
Post-failover replica resyncsUses a 1 MiB default repl-backlog-size and parallel-syncs > 1, causing full RDB resync storms after every failover.Sizes repl-backlog-size for 15+ minutes of peak writes, enables Valkey 8.0 dual-channel sync, and serializes parallel-syncs 1.
Client reconnect & DNS cachingRelies on DNS TTL propagation or expects application teams to manually restart pods when the primary IP changes.Wires Sentinel +switch-master pub/sub client pools or deploys sub-second HAProxy tcp-check routing probing role:master.
Rebooted empty-primary protectionEnables systemd Restart=always on cache nodes without disk persistence, risking a cluster-wide empty RDB wipe on reboot.Enforces hybrid AOF+RDB persistence or boot-time preStart scripts that attach recovered nodes as replicas before serving traffic.
Failover verification & drillsAssumes HA works because Sentinel daemons are running, discovering broken ACLs or firewall rules during a real outage.Runs scheduled failover and network-partition drills with measured application reconnect latency and written RCA reports.

Published initial-response targets by severity

Standard business hours: 03:30–12:30 UTC

Initial-response targets are 2 to 4 hours during covered business hours, Mon–Fri 09:00–18:00 IST (UTC+5:30). Where your service agreement includes 24x7 on-call, the initial-response target for P1 incidents is 15 minutes. Targets are not resolution guarantees.

SeverityInitial responseDefinitionChannel & coverage
P1 · Critical15 minutesProduction database down, primary unreachable, data corruption risk, or active failover event.24x7 on-call (under your service agreement) · Direct PagerDuty, Opsgenie, or phone page plus incident bridge
P2 · High30 minutesSevere latency spike, replication lag climbing rapidly, connection exhaustion, or degraded HA node.24x7 on-call (under your service agreement) · On-call alert plus shared Slack or Teams channel
P3 · Medium60 minutesNon-critical slow query regression, staging issue, backup warning, or replication check alert.24x7 on-call (under your service agreement) · Shared Slack or Teams channel or support ticket
P4 · Low90 minutesConfiguration question, index advisory, planned schema review, or general operational request.24x7 on-call (under your service agreement) · Shared Slack or Teams channel or support ticket

Our engineering team is in India; business-hours coverage is Mon–Fri 09:00–18:00 IST (UTC+5:30), and 24x7 on-call is available under your service agreement.

Initial-response targets, not resolution guarantees. The target for your engagement is set in your service agreement.

Interactive Sentinel quorum simulator

How Valkey Sentinel prevents split-brain during AZ outages

Compare a 3-AZ steady state against an unfenced split-brain partition and a hardened promotion guarded by min-replicas-to-write and PSYNC2 backlog sizing.

Valkey 8.0 Sentinel quorum & split-brain fencing simulator

Select a state to inspect Sentinel voting, primary write fencing, and replica re-parenting.

Sentinel Voters, HAProxy / SDK Discovery & Data Nodes

3 Sentinels across AZ-a, AZ-b, and AZ-c monitor the primary; HAProxy / Sentinel-aware clients route writes to AZ-a and reads to synced replicas.

Quorum 2/3 OK · Dual-channel PSYNC2 · Fencing armed
  • valkey-node-1 (AZ-a Primary)

    role:master · min-replicas-to-write 1 (lag <= 5s)

    ONLINE · 2 synced replicas connected

  • valkey-node-2 (AZ-b Replica + Sentinel-2)

    role:slave · replica-priority 100 · PSYNC2 active

    ONLINE · 0ms replication offset lag

  • valkey-node-3 (AZ-c Replica + Sentinel-3)

    role:slave · Offloaded RDB/AOF snapshots

    ONLINE · Quorum 3/3 reachable

In steady state, each Availability Zone hosts one Valkey data node and one Sentinel voter. The primary only accepts writes while at least one replica acknowledges replication within 5 seconds.

Representative production deliverable

How a Valkey high availability and Sentinel audit is delivered

Every HA engagement delivers hardened sentinel.conf and valkey.conf configurations, HAProxy health-check blocks, and chaos drill verification logs.

Sample finding · Unfenced primary split-brain & post-failover RDB resync stormValkey 8.0 Sentinel · 3-AZ Cluster

Finding

Unbounded isolated writes and 1 MiB replication backlog cause 4-minute recovery stalls

During an AZ network partition, the isolated primary kept accepting local writes for 95 seconds because min-replicas-to-write was 0. When Sentinel promoted a replica, the 1 MiB default repl-backlog-size forced a 48 GiB full RDB resync across remaining nodes.

End-to-end failover RTO240s to 6.8s
Split-brain write windowUnbounded to <= 5s

1. Hardened valkey.conf: Split-brain fencing & Valkey 8.0 dual-channel replication

valkey.conf · HA primary & replica directives

# Fence isolated primary if < 1 replica acknowledges within 5 seconds
min-replicas-to-write 1
min-replicas-max-lag 5

# Valkey 8.0 dual-channel diskless replication & 2 GiB PSYNC2 backlog
dual-channel-replication-enabled yes
repl-diskless-sync yes
repl-diskless-sync-delay 2
repl-diskless-load on-empty-db
repl-backlog-size 2gb
repl-backlog-ttl 3600

# Hybrid RDB + AOF persistence so rebooted nodes never start empty
appendonly yes
appendfsync everysec
aof-use-rdb-preamble yes

2. Hardened sentinel.conf (3-AZ quorum = 2) & quorum verification

sentinel.conf & valkey-cli verification

sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel monitor prod-valkey 10.40.1.10 6379 2
sentinel down-after-milliseconds prod-valkey 5000
sentinel parallel-syncs prod-valkey 1
sentinel failover-timeout prod-valkey 60000

# Verify 3-of-3 Sentinel quorum and test controlled failover
# valkey-cli -p 26379 SENTINEL CKQUORUM prod-valkey
# valkey-cli -p 26379 SENTINEL FAILOVER prod-valkey

Ways to work together

How to engage JusDB for Valkey high availability

Start with a free read-only HA and Sentinel quorum audit, scope a turnkey failover hardening project, or put your Valkey estate under ongoing 24x7 support under your service agreement.

Scoped HA engineering projectThis page

Harden Sentinel, fence split-brain, and drill failover.

A fixed-scope engagement delivering hardened Sentinel/Cluster configurations, HAProxy or SDK discovery tuning, and a rehearsed chaos failover drill.

Common HA scopes agreed in writing first

  • 3-AZ Sentinel quorum design
  • min-replicas-to-write fencing
  • Valkey 8.0 dual-channel sync
  • HAProxy & client pool routing
  • Live failover & partition drills
Free Valkey HA audit

One Valkey Sentinel or Cluster ring, audited read-only.

We inspect your Sentinel quorum placement, min-replicas-to-write fencing, replication backlog sizing, and client discovery config, then deliver a written risk report within one week.

What we check on your HA topology

  • Sentinel 3/5-node AZ distribution & quorum math
  • Split-brain write exposure & empty-reboot risk
  • PSYNC2 backlog sizing & client reconnect latency
Ongoing Valkey DBA & 24x7 support (where contracted)

Named engineers guarding your Valkey availability.

Proactive replication lag alerts, scheduled failover drills, backup restore verification, and incident response under a 24x7 support agreement.

  1. Page on-callavailable under a 24x7 support agreement
  2. 15 minP1 initial-response target
  3. Written RCAafter every P1

Valkey high availability topologies we support

Evaluation Vector
JusDB DBRE
In-House DBALegacy AgencyDeveloper Generalist
Sentinel Quorum & Multi-AZ Monitor Placement3 or 5-node odd-quorum Sentinel mesh distributed across independent availability zones with strict anti-affinity rules, co-located application monitoring, and zero single-point-of-failure topologies.Even-numbered Sentinel nodes deployed on same host or single AZ; network partition induces 50/50 vote split and total election deadlock.Single Sentinel instance monitoring cluster; any Sentinel crash or transient partition completely disables automated cluster failover.Sentinels deployed on same compute instances as Valkey nodes; node crash simultaneously eliminates monitor and quorum.
Asynchronous Replication Lag & Buffer SizingSub-second lag tracking via master_link_status and replication-offset deltas, paired with calibrated repl-backlog-size and client-output-buffer-limit tuning to prevent full resync loops.Default replication backlog buffers triggering full resync disk thrashing whenever network jitter exceeds 50ms.Ignores replication lag metrics until lagging replicas get promoted with massive data staleness during unexpected primary failures.Unmonitored async replication causing silent write divergence and unbounded memory growth under sustained load.
Split-Brain Prevention (min-replicas-to-write)Enforced min-replicas-to-write and min-replicas-max-lag constraints on primary nodes, isolating partitioned primaries and blocking phantom writes during network splits.Unconfigured min-replicas-to-write; partitioned primary continues accepting isolated writes that are permanently wiped upon healing.Misunderstands partition semantics; relies on client timeouts rather than engine-level fencing to prevent dual-primary states.Completely disables replica write guards, causing catastrophic data corruption when both partitions accept concurrent conflicting keys.
Automated Primary Failover & Client DiscoveryCalibrated down-after-milliseconds, fast Sentinel election, parallel replica resync, and client pub/sub +switch-master reconnection protocols, with failover timing tested against your RTO target.Default 30s failure detection combined with unhandled client connection caching resulting in 60s+ application write outages.Requires manual administrative intervention and manual DNS updates to redirect traffic following primary host failure.Hardcoded IP endpoints in client configuration; failover leaves application persistently attempting writes to dead node.
Cross-Region Disaster Recovery & RPO TargetsEngineered cross-region asynchronous replication standby, automated point-in-time RDB/AOF archival to object storage, and an RPO target agreed per workload.Daily snapshot backups stored locally without cross-region replication, resulting in 24-hour RPO data loss during regional outages.Manual off-site tar exports with unverified restore procedures; RTO exceeds 12 hours during catastrophic site failure.No off-site backup or cross-region DR plan; single cloud region failure results in total, irreversible data loss.
Chaos Engineering & Sentinel Leader Election DrillsAutomated chaos drills on an agreed cadence, simulating network partitions, hard node kills, and split-brain scenarios to validate quorum election and failover timing against your targets.Untested failover assumptions; first real failover test occurs during an unannounced 3 AM production crisis.Theoretical runbooks on paper without empirical failure testing; election timeouts fail under production connection stress.Zero chaos testing or disaster recovery drills; panic-driven ad-hoc reconfiguration during infrastructure incidents.

For procurement and infrastructure security teams

Mutual NDA first, read-only diagnostics, and rehearsed change windows

Failover and quorum changes are always rehearsed in staging first and executed during approved maintenance windows with your team on the bridge.Raguraman, Engineering Manager · Allen D, Lead Database SRE
  • Mutual NDA
  • Written statement of work
  • Named engineers
  • Least-privilege ACLs
  • NDA before access

    Signed before inspecting any Sentinel topologies, firewall rules, or replication configurations.

  • Written statement of work

    Covered clusters, RTO/RPO targets, chaos drill procedures, and fees agreed in writing before work starts.

  • Read-only by default

    Initial HA audits use read-only Valkey and Sentinel ACLs without touching production traffic.

  • Runbooks you keep

    Every sentinel.conf template, fencing script, and failover drill report stays with your team. Meet the JusDB team.

JusDB's security practices are aligned with ISO 27001 and SOC 2 Type II, and JusDB is pursuing both certifications (see Terms, section 12). Read the stated status.

Technical FAQ

Valkey high availability common questions

Answers on Sentinel vs Cluster HA, 3-of-5 quorum math, min-replicas-to-write split-brain fencing, Valkey 8.0 dual-channel replication, and client routing.

Still have a question?

Ask it on a scoping call. No production access needed.

Book a 30-min scoping call (opens Google Calendar in a new tab)

When should we use Valkey Sentinel versus Valkey Cluster for high availability?

Use Valkey Sentinel when your working dataset fits comfortably within the RAM of a single primary node (typically up to 64-128 GiB) and your application relies on multi-key operations (MGET, MSET, SUNION, RPOPLPUSH) or Lua scripts across arbitrary keys without {hashtag} constraints. Use Valkey Cluster when you need to shard datasets across multiple primaries (16,384 hash slots) or scale write throughput horizontally beyond a single node.

How many Valkey Sentinel nodes do we need, and where should they be placed?

You must always deploy an odd number of Sentinel nodes - minimum 3, or 5 for large production estates - distributed across at least 3 independent Availability Zones or physical failure domains with a quorum of 2 (for 3 Sentinels) or 3 (for 5 Sentinels). Never deploy only 2 Sentinels, and never place a majority of Sentinels in the same zone as the primary data node.

How do you prevent split-brain data loss in Valkey Sentinel?

Because Valkey uses asynchronous replication, a network partition can leave an isolated primary accepting writes from local clients while Sentinel promotes a replica in the majority partition. We bound this window by configuring min-replicas-to-write 1 and min-replicas-max-lag 5 (or 10) in valkey.conf. If the primary cannot reach at least one healthy replica within 5 seconds, it immediately stops accepting writes and returns -NOREPLICAS.

What is Valkey 8.0 dual-channel replication and how does it improve HA?

In older Redis versions, when a replica performs a full RDB resync after a network outage, the primary must hold all incoming live writes in an in-memory replica client output buffer until the RDB transfer completes, which often triggers OOM kills during recovery. Valkey 8.0 introduces dual-channel replication (dual-channel-replication-enabled yes), transferring the RDB snapshot and the incremental replication stream simultaneously so replicas recover faster with far lower memory pressure on the primary.

How should applications connect to a Valkey Sentinel topology?

We support two production patterns: (1) Sentinel-native client libraries (such as Lettuce, Jedis, ioredis, go-redis, or redis-py) configured with the full list of 3+ Sentinel endpoints so the driver subscribes to +switch-master events and routes directly to the current primary; or (2) a redundant HAProxy / Envoy TCP load-Balancing tier that runs active tcp-check probes verifying role:master via INFO replication every 1 second.

How do you prevent a rebooted primary with empty storage from wiping replicas?

If a Valkey primary runs without disk persistence (or loses its ephemeral volume) and restarts automatically as a primary, it will generate a fresh Replication ID with 0 keys and force all attached replicas to flush their memory. We prevent this by enabling hybrid RDB + AOF persistence (aof-use-rdb-preamble yes), configuring systemd/orchestration scripts so restarting nodes always boot in read-only replica mode first, and letting Sentinel manage promotion.

What values do you recommend for down-after-milliseconds and failover-timeout?

For production VPC and Multi-AZ cloud networks, setting down-after-milliseconds below 3000ms causes false failovers during routine GC pauses or BGSAVE forks. We typically calibrate down-after-milliseconds between 5000ms and 8000ms, parallel-syncs 1 (to protect the new primary after promotion), and failover-timeout 60000ms.

What are your response targets for Valkey high availability incidents?

Initial-response targets are 2 to 4 hours during covered business hours, Mon–Fri 09:00–18:00 IST (UTC+5:30). Where your service agreement includes 24x7 on-call, the initial-response target for P1 incidents is 15 minutes. Targets are not resolution guarantees.

Next step

Talk to an engineer about your Valkey high availability topology

Tell us how your Valkey or Redis Sentinel/Cluster nodes are deployed across zones today. The first 30-minute call covers your databases, current ownership, and required coverage. No production access is needed. Or request a free audit of one database instance, subject to availability: mutual NDA before read-only access, a prioritized report within one week after access is ready, and no obligation.

NDA before access · read-only by default · written scope first

Related Valkey Services

Explore more ways our Valkey experts can help with your database infrastructure.