A Valkey Cluster primary dies, and writes to its hash slots stop. With default settings, nothing happens for at least 15 seconds, because cluster-node-timeout is 15000 milliseconds and no replica may start an election until a majority of primaries agree the node is down. Then, with the default cluster-require-full-coverage yes, nodes answer CLUSTERDOWN The cluster is down until a replica takes over. Valkey changed the election part three times: 8.1 removed a vote limit, 9.0 scaled the election delay to the node timeout, and 9.1 lets the best replica start at once. Valkey 9.0 also added a SIGTERM handoff that the signals documentation does not mention yet. This post walks the timeline, version by version, and covers planned restarts.
- An unplanned Valkey Cluster failover is mostly detection: at least
cluster-node-timeout(15 seconds by default), then a FAIL agreement, then an election that takes well under a second for a healthy replica on 9.x. - 9.0 made the election delay
min(cluster-node-timeout / 30, 500)milliseconds, and 9.1 lets the best-ranked replica skip it. Target 9.1.3, which fixes a stale flag that could skip the delay wrongly. - Lowering
cluster-node-timeoutshortens detection but makes slow commands, fork pauses, and network blips look like failures, and it shrinks the replica validity window. - By default, SIGTERM waits up to
shutdown-timeout(10 seconds) for lagging replicas, runs a blocking SAVE when save points exist, and exits without a failover. On 9.0 and later,shutdown-on-sigterm failoverpromotes an in-sync replica first. - For maintenance you schedule, run
CLUSTER FAILOVERon the replica first. KeepFORCEfor a dead primary andTAKEOVERfor losing the majority.
Scope and review date: November 2026. This post covers self-managed Valkey Cluster on 8.0, 8.1, 9.0, and 9.1 (latest patches 8.0.12, 8.1.11, 9.0.7, and 9.1.3), with notes on the 9.2 release candidates. Timing rules were checked against the Valkey documentation and the cluster_legacy.c, server.c, and valkey.conf source of each tagged release on 10 November 2026. The commands were not run in a lab, and the timings below are computed from the source formulas, not measured, so rehearse a failover drill on staging.
What happens between a primary failing and clients writing again?
An automatic failover has five stages. A peer whose ping to the primary goes unanswered for longer than cluster-node-timeout flags it PFAIL. Once a majority of slot-owning primaries report that within twice the node timeout, it becomes FAIL. A replica waits an election delay, collects votes from a majority of primaries, takes over the slots, and broadcasts the change. Clients follow MOVED redirects or refresh their slot map.
The Valkey Cluster specification puts it as "NODE_TIMEOUT time plus a few more seconds required for a replica to get elected".
Valkey Cluster automatic failover, from failure to client redirect
The first two boxes are waiting; the election and promotion are short on 9.x
Where the time goes: steps 1 and 2 take at least the node timeout. On 9.1, step 4 is often 0 milliseconds for the best replica, so detection is almost the whole outage.
Valkey 9.0 added four node counters to CLUSTER INFO (PR #1910), so you can tell suspicion from agreement.
# Run against a node that is not the failing primary (Valkey 9.0+)
valkey-cli -h 10.0.1.12 -p 6379 CLUSTER INFO \
| grep -E 'cluster_state|nodes_pfail|nodes_fail|current_epoch'Interpretation: cluster_voting_nodes_pfail:1 means this node suspects a slot-owning primary, with no majority yet. cluster_voting_nodes_fail:1 with cluster_state:fail means the FAIL agreement happened. A higher cluster_current_epoch with cluster_state:ok means a replica won (CLUSTER INFO reference).
How does cluster-node-timeout decide when a primary is down?
Each node pings any peer it has not heard from for half the node timeout, and counts any bus traffic as proof of life. In 9.1.3, a peer becomes PFAIL when both the oldest unanswered ping and the last data received are older than cluster-node-timeout. The unanswered ping can go out up to half a timeout after the crash, so detection takes at least the timeout and often longer.
PFAIL is local. It becomes FAIL when reports arrive from a majority of slot-owning primaries, each no older than twice the node timeout. The failed primary still counts toward the cluster size, so in a three-shard cluster both survivors must report it. The node that reaches the quorum logs Marking node ... as failing (quorum reached). and broadcasts FAIL.
# Settings that shape detection; check every node, they are per-node
valkey-cli -h 10.0.1.12 CONFIG GET cluster-node-timeout
valkey-cli -h 10.0.1.12 CONFIG GET cluster-require-full-coverage
valkey-cli -h 10.0.1.12 CONFIG GET cluster-replica-validity-factorInterpretation: the defaults in valkey.conf for 9.1.3 are 15000, yes, and 10. Keep the timeout identical across nodes. With full coverage on, a node that sees any slot owned by a FAIL node switches to cluster_state:fail and rejects keyed commands for every slot.
# What a client sees on a healthy shard while the failed shard has no primary
valkey-cli -c -h 10.0.1.12 SET user:42 active
# (error) CLUSTERDOWN The cluster is downInterpretation: the error string is from clusterRedirectClient() in 9.1.3. cluster-require-full-coverage no keeps the other shards serving during the failover, and cluster-allow-reads-when-down yes keeps reads going. Both trade consistency for availability.
A primary cut off from the majority also switches to fail after the node timeout and refuses writes. That bounds how long a partitioned primary accepts writes that will later be lost, so the timeout sets this window too.
How long is the election delay, and how does replica rank change it?
After FAIL, each replica of the failed primary schedules its election. The cluster specification still prints the older rule: "500 milliseconds + random delay between 0 and 500 milliseconds + REPLICA_RANK * 1000 milliseconds". That is what 8.0 and 8.1 run. The 9.1.3 source computes it differently.
# Election delay in Valkey 9.0 and 9.1, from clusterHandleReplicaFailover()
# delay = min(cluster-node-timeout / 30, 500) ms
# start = now + delay + random(0 .. delay)
# start += replica_rank * (2 * delay)
# start += 500 ms if this replica's offset is 0 (never synced)
# start += failed_primary_rank * delay (several primaries down)
# 9.1 only: the best-ranked replica starts immediately when every
# replica of the shard has reported the primary as FAIL.
# Votes: wait max(2 * node-timeout, 2000) ms, retry after twice that.Interpretation: rank is the number of sibling replicas with a higher replication offset, so the most up-to-date replica is rank 0. At the default timeout the fixed delay is still 500 ms and rank 1 waits an extra second. The bigger cost is a split vote: the election expires after 30 seconds and the retry waits for 60.
The failed-primary term arrived in 8.1 (#1018 in the 8.1.0 release notes) so replicas of several failed primaries do not collide. The never-synced penalty is from PR #2449; why an empty replica can still win is covered in replacing a failed Valkey Cluster node. Each replica logs its decision:
grep -E 'FAIL message received|best ranked replica|Start of election delayed|Failover election won|Failover auth (granted|denied)' \
/var/log/valkey/valkey.logInterpretation: on the replica, the gap from the FAIL line to Failover election won is the election time. On primaries, Failover auth denied lines give the reason a vote was refused.
Replica log and CLUSTER INFO during a 9.1 automatic failover
Illustrative output: log messages match the 9.1.3 source, node IDs, times, and counts are invented
Read it this way: the old primary still counts in cluster_nodes_fail, but it owns no slots now, so cluster_voting_nodes_fail is back to 0. The 15 seconds before the first line are not in this log.
What did Valkey 8.1, 9.0 and 9.1 change in failover timing?
PR #1356 (8.1.0, backported to 8.0.10) removed the rule that a primary will not vote for another replica of the same primary for twice the node timeout. PR #2449 (9.0.0) scaled the delay to the timeout, PR #2227 (9.1.0) lets the best-ranked replica start at once, and PRs #1091 and #2292 (9.0.0) added failover on shutdown.
Failover timing by Valkey version, at the default 15000 ms node timeout unless noted
Switch versions to see which delays apply to a replica after its primary is marked FAIL
Valkey 8.0 and 8.1
| Timing rule | Valkey 8.0 and 8.1 |
|---|---|
| Fixed election delay | 500 ms |
| Fixed delay at a 3000 ms node timeout | 500 ms |
| Extra delay per replica rank | 1000 ms |
| Replica that has never synced | No extra delay |
| Best-ranked replica | Waits the full delay |
| Vote limit of 2x node timeout per primary | Removed in 8.1.0 and 8.0.10 |
| Failover on SIGTERM or SHUTDOWN | Not available |
Valkey 9.0
| Timing rule | Valkey 9.0 |
|---|---|
| Fixed election delay | 500 ms |
| Fixed delay at a 3000 ms node timeout | 100 ms |
| Extra delay per replica rank | 2 x fixed delay (1000 ms at default) |
| Replica that has never synced | Extra 500 ms |
| Best-ranked replica | Waits the full delay |
| Vote limit of 2x node timeout per primary | Removed |
| Failover on SIGTERM or SHUTDOWN | shutdown-on-sigterm failover, SHUTDOWN FAILOVER |
Valkey 9.1
| Timing rule | Valkey 9.1 |
|---|---|
| Fixed election delay | 500 ms |
| Fixed delay at a 3000 ms node timeout | 100 ms |
| Extra delay per replica rank | 2 x fixed delay (1000 ms at default) |
| Replica that has never synced | Extra 500 ms |
| Best-ranked replica | Starts at once if all replicas report FAIL (fixed in 9.1.3) |
| Vote limit of 2x node timeout per primary | Removed |
| Failover on SIGTERM or SHUTDOWN | shutdown-on-sigterm failover, SHUTDOWN FAILOVER |
Upgrade note: before 9.1.3, a stale "my primary failed" flag could let a later failover skip the delay before offsets were exchanged (PR #4533, 9.1.3 release notes).
Inside the first panel, 8.1 added the failed-primary ordering and cluster-manual-failover-timeout; 8.0 has a fixed 5000 ms manual failover timeout. A node timeout below 30 ms made the 9.x delay code undefined until 9.0.5 and 9.1.1 (#3941).
valkey-cli -h 10.0.1.21 INFO server | grep '^valkey_version'Interpretation: run it on every replica. The delay is computed on the replica, so mid-upgrade a shard can fail over with 8.x timing on one replica and 9.1 timing on another.
9.2 release candidate: 9.2.0-rc1 adds a FAILOVER_AUTH_NACK message (PR #3833) so a replica that cannot win a split vote retries at once, and rc2 fixes three edge cases in it. It is not GA behaviour yet.
Should you lower cluster-node-timeout to fail over faster?
The node timeout is the only setting that shortens the long part of an unplanned failover, and most other cluster timers are multiples of it. The table applies the 9.1.3 formulas; these are computed bounds, not measurements.
| Effect of cluster-node-timeout | 15000 (default) | 5000 | 3000 |
|---|---|---|---|
| Minimum detection time | 15 s | 5 s | 3 s |
| Fixed election delay (9.0, 9.1) | 500 ms | 166 ms | 100 ms |
| Election expires after | 30 s | 10 s | 6 s |
| Retry after a failed election | 60 s | 20 s | 12 s |
| Replica too stale to fail over after | 160 s | 60 s | 40 s |
| Ping to each idle peer at least every | 7.5 s | 2.5 s | 1.5 s |
The validity row uses the valkey.conf formula (node-timeout * cluster-replica-validity-factor) + repl-ping-replica-period with the defaults of 10 and 10 seconds. A replica whose link to the primary was down longer than that refuses an automatic failover, so a shorter timeout makes a shard more likely to stay down after a long network problem.
The main risk is false failovers. Bus replies come from the main thread, so a stall longer than the timeout looks like a dead node: a slow KEYS or Lua script, a large key deletion, a slow fork, or a VM pause. The cluster tutorial's example uses 5000, a reasonable value to test first.
# Find main-thread stalls before shortening the timeout
valkey-cli -h 10.0.1.11 CONFIG SET latency-monitor-threshold 100
valkey-cli -h 10.0.1.11 LATENCY LATESTInterpretation: each entry shows an event name, when it last fired, and its latest and maximum latency in milliseconds. A maximum anywhere near half your proposed timeout means the new value would produce failovers you did not want. Fix the stall first.
# Apply the same value on every node, then persist it
for h in 10.0.1.11 10.0.1.12 10.0.1.13 10.0.1.21 10.0.1.22 10.0.1.23; do
valkey-cli -h "$h" CONFIG SET cluster-node-timeout 5000
valkey-cli -h "$h" CONFIG REWRITE
doneInterpretation: the setting changes at runtime, per node, and CONFIG REWRITE persists it. On a managed service or an operator, change it through the parameter group or custom resource instead.
What does SIGTERM do to a Valkey primary by default?
SIGTERM and SIGINT run the same sequence as SHUTDOWN with no arguments. Read from the SHUTDOWN command reference and finishShutdown() in the 9.1.3 source, the default does this:
- If any replica is behind, pause client writes and wait up to
shutdown-timeout(10 seconds) for replicas to catch up. - Kill any background RDB save or AOF rewrite child. If the first AOF rewrite after enabling AOF is still running, refuse to exit.
- Flush and fsync the AOF if AOF is on.
- Run a blocking
SAVEif at least one save point is configured. If it fails, refuse to exit. - Save
nodes.conf, close the sockets, and exit. No failover happens.
On a primary, step 5 means the shard waits for the full detection timeline. Steps 1 and 4 also block writes before the exit. Under Kubernetes, the default 30-second termination grace period ends in SIGKILL, and a SIGKILL during the SAVE leaves the previous RDB file as the newest on disk.
valkey-cli -h 10.0.1.11 CONFIG GET shutdown-on-sigterm
valkey-cli -h 10.0.1.11 CONFIG GET shutdown-timeout
valkey-cli -h 10.0.1.11 CONFIG GET save
valkey-cli -h 10.0.1.11 CONFIG GET appendonlyInterpretation: default plus a non-empty save means every SIGTERM runs a blocking save. rdb_last_bgsave_time_sec in INFO persistence gives a rough idea of its length.
The Signal handling page describes SIGTERM as a fixed sequence and does not mention shutdown-on-sigterm, shutdown-on-sigint, or the 9.0 failover and safe values. Only valkey.conf and the SHUTDOWN reference document them.
How do shutdown-on-sigterm failover and SHUTDOWN FAILOVER hand off a primary?
Valkey 9.0 added a failover value for shutdown-on-sigterm and shutdown-on-sigint, and a FAILOVER modifier for SHUTDOWN (PR #1091, renamed into this form by PR #2292). It applies "in cluster mode, when shutting down a primary" (valkey.conf). In 9.1.3, the primary picks an online replica that has acknowledged the full offset and sends it CLUSTER FAILOVER FORCE, so the election starts at once.
# valkey.conf on every node (9.0 or later); takes effect on primaries only
shutdown-timeout 10
shutdown-on-sigterm failover
shutdown-on-sigint failoverInterpretation: values combine (failover nosave is valid; save nosave is not). Keep shutdown-timeout above 0: the in-sync check runs after the wait, and without it a replica that has not acknowledged the latest offset is skipped.
Three conditions in the source decide whether the handoff happens:
- Every replica runs 9.0 or later. Otherwise the primary logs
Unable to perform auto failover on shutdown since there are legacy replicas.and exits without one, which matters mid-upgrade from 8.x. - At least one replica is fully caught up. Otherwise the log says
Unable to find a replica to perform the auto failover on shutdown. - The order is fixed. The failover is sent after the AOF flush and any blocking SAVE, so a slow save delays the handoff. If replicas hold the persistent copies,
failover nosaveremoves that wait; test the restart path first.
# Manual equivalents on a 9.x cluster primary
valkey-cli -h 10.0.1.11 SHUTDOWN SAFE # refused: primary owns slots
valkey-cli -h 10.0.1.11 SHUTDOWN FAILOVER # promote an in-sync replica, then exit
# Primary log on success:
# Perform auto failover to replica <node-id> on shutdown.Interpretation: SAFE (PR #2195) refuses to stop a primary that owns slots. In 9.1.3 that check runs before the failover step, so SHUTDOWN SAFE FAILOVER on a primary is refused, not handed off. Do not combine safe and failover in shutdown-on-sigterm on a possible primary: it would ignore SIGTERM until SIGKILL.
Which CLUSTER FAILOVER variant should you use: default, FORCE or TAKEOVER?
CLUSTER FAILOVER is sent to a replica and promotes it in place of its primary. The CLUSTER FAILOVER reference describes three forms.
| Form | What it skips | Data loss window | Use it when |
|---|---|---|---|
CLUSTER FAILOVER | Failure detection only | None: the primary pauses writes and the replica catches up first | Planned maintenance with a healthy primary |
CLUSTER FAILOVER FORCE | The handshake with the primary | Writes the replica had not received | The primary is unreachable but a majority of primaries is up |
CLUSTER FAILOVER TAKEOVER | The handshake and the vote | As FORCE, plus epoch conflicts | A majority of primaries is gone, such as a data center switch |
In the default form, the primary pauses writes, the replica catches up and runs an election with no delay. If it cannot win in time, both give up after cluster-manual-failover-timeout (5000 ms default, since 8.1). In 9.1.3 the primary lifts its write pause when its own copy of that timer expires, with a hard cap at twice the value.
# Run on the replica you want to promote, not on the primary
valkey-cli -h 10.0.1.21 CLUSTER FAILOVER
# Then confirm the role change on the same node
valkey-cli -h 10.0.1.21 ROLEInterpretation: OK only means the failover was scheduled; ROLE returning master first confirms it. If the replica logs Manual failover timed out., the old primary resumes writes and nothing changed. Raising cluster-manual-failover-timeout gives a slow replica longer but lengthens the worst-case write pause.
TAKEOVER generates a config epoch without consensus, and the documentation says it "should be used with care". Fence the old primaries first.
How do you restart a cluster primary without waiting for node timeout?
For a scheduled restart, hand the slots over first, so the write pause is a manual failover instead of the detection timeline. shutdown-on-sigterm failover covers the restarts you do not schedule, such as pod evictions.
Planned handoff of a Valkey 9.x cluster primary
Seven steps; steps 4 and 7 pause writes to this shard for the handoff
-
Check versions and replica sync
Every replica should be online with
lag=0or 1 and run the same 9.x patch as the primary.valkey-cli -h 10.0.1.11 INFO replication \ | grep -E '^slave[0-9]+:' -
Set the shutdown safety net
Covers a SIGTERM that arrives outside this runbook.
valkey-cli -h 10.0.1.11 CONFIG SET \ shutdown-on-sigterm failover valkey-cli -h 10.0.1.11 CONFIG REWRITE -
Confirm every primary knows the replica
A replica can only win if a majority of primaries list it as a replica of this shard.
valkey-cli -h 10.0.1.12 CLUSTER REPLICAS \ <primary-node-id> -
Promote the replica
Write pauseWrites pause until the replica catches up and wins, or the timeout ends.
valkey-cli -h 10.0.1.21 CLUSTER FAILOVER -
Verify the new roles
The promoted node reports master; the old primary reports slave and follows it.
valkey-cli -h 10.0.1.21 ROLE valkey-cli -h 10.0.1.11 ROLE -
Restart the old primary as a replica
Writes are unaffected. Wait for the link before the next shard.
sudo systemctl restart valkey valkey-cli -h 10.0.1.11 INFO replication \ | grep master_link_status -
Fail back if placement matters
Write pauseOnly if the original node must be primary again, for example for zone placement.
valkey-cli -h 10.0.1.11 CLUSTER FAILOVER
Rollback point: until step 4 succeeds, nothing has changed; a timed-out manual failover leaves the old primary in charge. After step 4, going back is another CLUSTER FAILOVER in the other direction, as in step 7.
If the primary is already down, the default form returns Master is down or failed, please use CLUSTER FAILOVER FORCE. For the choice between this and Sentinel-style failover, see Valkey Cluster vs Sentinel.
Frequently asked questions
How long does a Valkey Cluster failover take?
An unplanned failover takes at least cluster-node-timeout, 15 seconds by default, before the primary is even suspected. On Valkey 9.x the election adds little: the fixed delay is 500 ms or less, and on 9.1 the best replica can start at once. A planned CLUSTER FAILOVER skips detection. Measure yours with a failover drill.
What is the default cluster-node-timeout in Valkey?
The default is 15000 milliseconds in every current Valkey release, including 9.1.3. It can be changed at runtime with CONFIG SET and applies per node, so set the same value everywhere and persist it with CONFIG REWRITE. The cluster tutorial's example uses 5000.
Does SIGTERM trigger a failover in Valkey Cluster?
Not by default. A primary waits for lagging replicas, saves if save points exist, and exits, so its replicas wait for the full node timeout. From Valkey 9.0, shutdown-on-sigterm failover makes a cluster primary promote a fully synced replica before exiting, as long as every replica runs 9.0 or later.
What is the difference between CLUSTER FAILOVER FORCE and TAKEOVER?
FORCE skips the handshake with the primary, so it works when the primary is unreachable, but still needs votes from a majority of primaries. TAKEOVER also skips the vote and assigns itself a new config epoch. Use TAKEOVER only when the majority is gone, because it breaks the last-failover-wins rule.
Can I set cluster-node-timeout below one second?
Valkey accepts it, but any main-thread stall longer than the timeout, such as a slow script or a fork pause, then looks like a dead node and can trigger a failover. It also shortens the replica validity window. Test 5000 ms first and measure stalls with LATENCY LATEST.
Key takeaways
- An unplanned failover is detection plus a short election: plan around
cluster-node-timeout, and expectCLUSTERDOWNon every node whilecluster-require-full-coverageisyes. - Run 9.1.3 or later to get the immediate election for the best replica without the stale-flag bug; 9.0 already scales the delay to the node timeout.
- Lower the node timeout only after measuring main-thread stalls, and remember it also shortens the election expiry, retry, and replica validity windows.
- Set
shutdown-on-sigterm failoveron 9.x cluster nodes, keepshutdown-timeoutabove 0, and never combine it withsafeon a node that may be a primary. - For maintenance you schedule, run
CLUSTER FAILOVERon the replica first and restart the old primary as a replica.
Tune Valkey Cluster failover with JusDB
JusDB's Valkey Cluster team can review your node timeout, coverage, and shutdown settings against your write-availability target, check client retry behaviour during a failover, and write a rolling restart runbook for your topology, whether it runs on VMs or Kubernetes.
Bring your Valkey versions, current CONFIG GET output, shard layout, and the restart windows you have. Your team approves every production change.