Replacing a Failed Valkey Cluster Node Without Wiping the Shard

A Valkey Cluster node dies, you add a replacement replica, and before it finishes its initial sync the shard's primary fails too. The new replica has a replication offset of 0, yet nothing in the election stops it. The primaries vote for it, it promotes itself, and the shard comes back online serving zero keys. CLUSTER INFO reports cluster_state:ok, and when the old primary returns it rejoins as a replica of the empty node and throws away its own copy. Valkey 9.0, 9.0.5 and the 9.2 release candidates each narrow this window. This guide explains the failure mode, how to remove a dead node ID, and a replacement runbook that checks offsets before any failover.

Scope and review date: November 2026. This post covers self-managed Valkey Cluster on 8.0, 8.1, 9.0 and 9.1, with notes on the 9.2 release candidates (9.2.0-rc2, published 4 November 2026, is the newest). Versions, defaults and log messages were checked against Valkey's documentation, GitHub release notes and source on 10 November 2026. The commands were not run in a lab, so rehearse them on a staging cluster first.

How can an empty replica win a Valkey Cluster election?

When a primary is marked FAIL, each of its replicas schedules an election and asks every slot-owning primary for a vote. The Valkey Cluster specification is blunt about how votes are granted: "Primaries make no effort to select the best replica in any way." A primary votes for the first replica that asks, as long as that replica's primary is in FAIL and it has not already voted in this round.

Only timing favours the replica with the most data: replicas rank themselves by replication offset, and the best-ranked one asks first. That ranking is best effort. A replica that joined seconds ago may not know the others yet, and a lone surviving replica has nobody to outrank it. The pull request that added the 9.2 guard, #4425, says empty-replica promotion "is especially likely when it is the only surviving replica". That is exactly the state a shard is in halfway through a node replacement.

Each election attempt is logged with the offset the replica used to rank itself:

# On each replica (or former replica) of the shard that failed over
grep -E "Start of election delayed|Failover election won|Currently unable to failover" \
    /var/log/valkey/valkey-server.log | tail -n 5

Interpretation: in 9.1 the line reads "Start of election delayed for N milliseconds (rank #R, primary rank #P, offset O)". If the node that logged "Failover election won: I'm the new primary." shows offset 0, the shard is now empty. The log path follows your logfile setting.

Why can restarting a node with an empty data directory wipe a shard?

Valkey Cluster tells nodes apart by node ID, not by address. The 40-character ID is generated the first time a node starts and saved in its cluster config file (nodes.conf by default). It lasts as long as that file does, unless you run CLUSTER RESET HARD.

The replication documentation describes the danger: a primary "that crashes and comes back as the same instance with an empty dataset" makes its replicas "faithfully replicate that empty dataset and discard their own copies". Valkey Cluster avoids this when a replacement starts without cluster configuration: it gets a fresh ID and joins as a new, empty node, even on the old IP address, so the surviving replicas keep their data.

The risk sits between those cases: a node that keeps its nodes.conf but loses its data, for example a Kubernetes pod whose volume keeps nodes.conf while persistence is off.

valkey-cli -h 10.0.1.14 CONFIG GET appendonly save dir cluster-config-file

Interpretation: if appendonly is no and save is empty, the node writes no data to disk. Its nodes.conf (relative to dir unless the path is absolute) is then the only thing that survives a restart, and a fast restart in place brings back the old ID with no data.

What did Valkey 9.0, 9.1 and the 9.2 release candidates change?

Four changes touch this failure mode; the cluster tutorial mentions none of them.

• 9.0: a delay tied to the node timeout, plus a penalty for new replicas. PR #2449 sets the fixed election delay to min(500, node_timeout / 30) ms, which is 500 ms at the default 15,000 ms timeout. It also adds 500 ms for a replica at offset 0: "This avoids the scenario where a new replica with no data wins the failover." The cluster spec page still shows the older formula.
• 9.0.5 and 9.1.0: stale cluster packets. PR #2811 fixes a race where a replica acted on an old packet, re-attached itself to the failed primary and won an election as an empty primary. The PR says the 8.x backport was skipped.
• 9.1: the best-ranked replica skips the delay. PR #2227 lets the best-ranked replica start at once. In the 9.1.3 source, a replica at offset 0 never takes this shortcut.
• 9.2 release candidates: if-empty. Covered in the next section.

valkey-cli -h 10.0.2.15 INFO server | grep valkey_version
valkey-cli -h 10.0.2.15 CONFIG GET cluster-replica-no-failover \
    cluster-replica-validity-factor cluster-node-timeout

Interpretation: read valkey_version, not the redis_version:7.2.4 compatibility field. Check every node: the replica that runs the election is the one whose version counts.

Should you set cluster-replica-no-failover to if-empty?

With if-empty, a replica refuses to start an automatic failover while its replication offset is 0, because it "has never completed an initial sync with its primary", in the words of the 9.2.0-rc2 valkey.conf. "Empty" means no data received from the primary, not zero keys: a replica fully synced with an empty primary has a non-zero offset and can still fail over. The default stays no.

The trade is availability for data. If the only replica left is empty, the shard stays down until a replica with data is available, the old primary returns, or you fail over manually. For a cache you can refill, an empty shard may be acceptable; for data you cannot rebuild, a shard that stays down is easier to recover.

# valkey.conf on every replica: 9.2 release candidates only
cluster-replica-no-failover if-empty

Interpretation: the setting can also be changed with CONFIG SET. When it blocks a failover, the replica logs "Currently unable to failover: Replication offset is 0 and no data has been received from the primary." That makes the refusal visible instead of silent.

Manual failovers bypass if-empty, so CLUSTER FAILOVER FORCE or TAKEOVER sent to an empty replica still promotes it. Also check cluster-replica-validity-factor (default 10): it stops a replica that has been disconnected longer than node timeout times the factor plus repl-ping-replica-period. Setting it to 0 drops that data-age check; Valkey's own if-empty test sets it to 0 so the stalled replica becomes a candidate immediately.

How do you check a replica's offset before a failover?

Two commands give you the numbers. CLUSTER SHARDS returns each node's replication-offset and a health of online, fail or loading. In the 9.1.3 source, a replica reports loading exactly when its replication offset is 0, the same condition if-empty tests. INFO replication shows the link state and the offsets on both sides.

# Any replica still at offset 0 shows up as "loading"
valkey-cli -h 10.0.2.15 CLUSTER SHARDS | grep -A1 -E 'replication-offset|health'

Interpretation: every replica you might fail over to should show online and an offset close to its primary's. A replica that shows loading is the one that must not be promoted.

On a busy primary both numbers keep moving, so also check lag on the primary's slave0 line. During the initial sync, master_link_status stays down.

How do you remove an unreachable node from the cluster?

Until recently, valkey-cli --cluster del-node loaded only reachable nodes and failed on a dead one with [ERR] No such node ID. PR #3209 fixed that in 8.0.10, 8.1.8, 9.0.5 and 9.1.0. The cluster tutorial still tells you not to use del-node for a failed node, which is out of date for those versions.

# Valkey 8.0.10, 8.1.8, 9.0.5, 9.1.0 and later
DEAD_ID=9f2b61c0de5e8a7a4d0c3e1b2f6a9d8c7b5e4f30   # 40-character ID from CLUSTER NODES
valkey-cli --cluster del-node 10.0.2.15:6379 "$DEAD_ID"

Interpretation: on a fixed version the output warns that it "Could not connect" to the dead node and is "unable to send CLUSTER RESET", then ends with [OK] Node ... removed from the cluster. That is expected. If it refuses with "is not empty! Reshard data away and try again.", the node still owns slots: no failover happened, and you need the original node back or a forced failover to a replica that has data.

On older patches, send CLUSTER FORGET to every remaining node yourself. Each node bans the forgotten ID for 60 seconds so that gossip from nodes that still remember it cannot re-add it. Since 8.0 that window is cluster-blacklist-ttl, in seconds, and it can be changed at runtime.

# Older patch levels: forget the dead ID on every node in one pass
valkey-cli --cluster call 10.0.2.15:6379 CLUSTER FORGET "$DEAD_ID"

# Forgetting node by node? Widen the ban window first, then restore it
valkey-cli --cluster call 10.0.2.15:6379 CONFIG SET cluster-blacklist-ttl 300

Interpretation: --cluster call skips nodes it cannot reach, and every other node should reply OK. Forget only a node that no longer owns slots, and note that a replica refuses to forget its own current primary. On 7.2 the window is fixed at 60 seconds.

# Every node should print 0 once the ID is gone everywhere
for h in 10.0.1.11 10.0.1.12 10.0.2.11 10.0.2.12 10.0.2.15; do
  echo "$h $(valkey-cli -h "$h" CLUSTER NODES | grep -c "$DEAD_ID")"
done

Interpretation: if a count returns to 1 later, some node re-learned the ID through gossip; forget it everywhere again. Before reusing the old host, delete its nodes.conf or run CLUSTER RESET HARD on it.

Which patch fixes truncated nodes.conf files?

Replacing a node means fresh nodes.conf writes on every member as the topology changes. PR #4489, released in 9.1.3, 9.0.7, 8.1.11, 8.0.12 and 7.2.15 in late October 2026, fixes three problems in how that file was saved:

• A write error could go unnoticed, so a truncated nodes.conf was published. The node then panicked while loading it and would not start.
• After the first save, the startup lock no longer covered nodes.conf, so a second process could start on a config already in use.
• On SMB file systems such as Azure Files, every save failed and the process exited, which shows up as a pod crash loop.

The lock now lives in a sibling file with .lock appended. The 9.1.3 valkey.conf warns that it must stay creatable, which matters for Kubernetes subPath mounts of only the config file and for directory-cleaning sidecars.

ls -l /var/lib/valkey/nodes*.conf*

Interpretation: a fixed version shows the config file and its .lock sibling. Valkey 9.1 also added cluster-config-save-behavior: the default sync exits when a save fails, while best-effort logs and keeps running, at the risk of a stale config after a restart.

What is the safe order for replacing a failed node?

The runbook covers the common case: a node is gone for good and its shard is one replica short. If the dead node was the primary, step 2 is the one that matters.

Frequently asked questions

Key takeaways

Replace Valkey Cluster nodes with JusDB

JusDB's Valkey Cluster team can review your replica placement, persistence and failover settings against the failure modes in this post, and write a node-replacement runbook that fits your hosting, from VMs to Kubernetes.

Bring your Valkey versions, your valkey.conf cluster settings, and how nodes are restarted or rescheduled today. Your team approves every production change.

Related reading

• Valkey 9.1: database ACLs, Lua and I/O threading, and how to upgrade
• Valkey and Redis operators for Kubernetes compared
• Valkey performance tuning: memory, persistence, and cluster config
• Valkey documentation: Cluster specification