Resharding Valkey 9 with Atomic Slot Migration: A Production Runbook

Resharding a Valkey Cluster used to mean moving keys a batch at a time with MIGRATE. While a slot was half moved, clients chased -ASK redirects, multi-key commands failed with -TRYAGAIN, and one large sorted set could stall the move. Valkey 9.0 added atomic slot migration: the source primary streams whole slots to the target and hands ownership over in one step, driven by CLUSTER MIGRATESLOTS, CLUSTER GETSLOTMIGRATIONS and CLUSTER CANCELSLOTMIGRATIONS. It still has costs. The source forks, buffers writes to the moving slots, and pauses writes briefly at the end, and before 9.0.7 and 9.1.3 it could apply writes to the wrong database when cluster-databases is above 1. This runbook covers the checks, monitoring, cancelling and rollback.

Scope and review date: November 2026. This post covers self-managed Valkey Cluster 9.0 and 9.1 (latest patches 9.0.7 and 9.1.3). Commands, defaults and messages were checked against the valkey.io documentation, the release notes and the 9.1.3 source on 10 November 2026. They were not run in a lab, so rehearse every step on staging.

How is atomic slot migration different from MIGRATE-based resharding?

Legacy resharding is driven from outside the cluster, one slot at a time. The Valkey deep dive on atomic slot migration lists the steps that valkey-cli --cluster reshard automates:

# Legacy slot migration, repeated for every slot
CLUSTER SETSLOT <slot> IMPORTING <source-id>    # on the target
CLUSTER SETSLOT <slot> MIGRATING <target-id>    # on the source
CLUSTER GETKEYSINSLOT <slot> 10                 # on the source, until empty
MIGRATE <target-ip> 6379 "" 0 60000 KEYS k1 k2  # on the source, per batch
CLUSTER SETSLOT <slot> NODE <target-id>         # on the target, then the source

Interpretation: every batch costs network round trips, and valkey-cli defaults to 10 keys per batch. While the slot is split, the source answers -ASK for keys already moved, and a multi-key command spanning both nodes gets -TRYAGAIN. A large collection moves as one serialized payload.

Atomic slot migration moves the work into the server. Per the atomic slot migration topic page, the source authenticates to the target much as a replica would, forks a child that writes the slots as a stream of commands (collections travel as chunks of elements), then streams the writes made meanwhile. When little is left in flight, it pauses writes, the target catches up and claims the slots, and the source deletes its copy.

What do clients see during an atomic slot migration?

Until the handoff, nothing changes. The source still owns the slots and serves every command for them, so there are no -ASK redirects and no -TRYAGAIN errors. On the target, importing slots are hidden from KEYS, SCAN, DBSIZE and INFO on the primary and its replicas.

At the handoff, the source pauses client writes with the same mechanism as a manual failover; reads continue. Once the target broadcasts ownership, the source deletes the keys and releases the paused clients, which follow MOVED. Valkey's benchmark shows a sharper but shorter throughput dip than legacy migration.

valkey-cli -h 10.0.1.11 CONFIG GET cluster-manual-failover-timeout
valkey-cli -h 10.0.1.11 CONFIG GET slot-migration-max-failover-repl-bytes

Interpretation: the 9.1.3 valkey.conf says cluster-manual-failover-timeout (default 5000 ms) also bounds migration finalization; in the 9.1.3 source, writes stay paused for at most twice that, 10 seconds at the default. slot-migration-max-failover-repl-bytes (default 0) starts the pause only when the source's buffer is empty. Raising it lets a write-heavy slot reach the pause sooner, at the cost of a longer pause.

What must be true before you run CLUSTER MIGRATESLOTS?

The CLUSTER MIGRATESLOTS reference lists it since 9.0.0. The 9.1.3 source rejects it before anything starts in these cases:

• Not the source primary. A replica answers "Slot migration can only be used on primary nodes.", and a node that does not own the range answers "Slots are not served by this node."
• A legacy migration is open. Any slot in MIGRATING or IMPORTING state returns "Slots are being manually migrated" or "Slots are being manually imported".
• A module has not opted in. Since 9.0.0-rc2, modules must declare support, or the command fails with "The module ... does not support atomic slot migrations."

Every node involved must also run 9.0 or later, since the target must speak the internal CLUSTER SYNCSLOTS protocol, and 9.0.7 or 9.1.3 is the floor for the fixes below.

valkey-cli --cluster call 10.0.1.11:6379 INFO server | grep valkey_version
valkey-cli -h 10.0.1.11 CLUSTER NODES | grep -E '\[[0-9]+-(<|>)-'
valkey-cli --cluster call 10.0.1.11:6379 MODULE LIST

Interpretation: the second command must print nothing. A [slot->-id] or [slot-<-id] entry is an unfinished legacy move; close it with valkey-cli --cluster fix first.

The migration user needs write access

The source authenticates to the target as the primaryuser, and the target applies the data as ordinary commands under that user's ACL. A replica-only ACL works on empty slots and fails once a slot holds data. The topic page and the ACL requirements post give this rule string:

# On every node; rule order matters
ACL SETUSER replicator on >change-me \
  +@write ~* -@dangerous +ping +select +psync +replconf \
  +cluster|syncslots -flushall -flushdb -restore -restore-asking

# Prove it on the target before the first move
valkey-cli -h 10.0.1.21 ACL DRYRUN replicator SET probe 1

Interpretation: OK means the user may write. If not, the target logs -NOPERM User replicator has no permissions to run the 'set' command, while the source reports only "Connection lost to target. Check CLUSTER GETSLOTMIGRATIONS on the target node for more information." The operator's user also needs the alldbs rule for MIGRATESLOTS and CANCELSLOTMIGRATIONS.

How much headroom does an atomic slot migration need?

When one of these runs short, the migration usually fails and cleans up, but a fork or memory squeeze can slow live traffic first.

• Source fork. The snapshot child costs copy-on-write, reported as cow_size, and the job waits in waiting-to-snapshot while a BGSAVE or AOF rewrite child runs. Our fork latency post covers the fork cost itself.
• Source output buffer. Writes made during the snapshot queue under client-output-buffer-limit replica (9.1.3 default 256mb 64mb 60). The topic page says to size both limits for them; an overrun fails the job.
• Target memory. The target primary and each of its replicas must hold the incoming slots. An import that runs out fails with "Ran out of memory (OOM) during slot import".

valkey-cli -h 10.0.1.11 CLUSTER SLOT-STATS SLOTSRANGE 0 511
valkey-cli -h 10.0.1.11 CLUSTER SLOT-STATS ORDERBY key-count LIMIT 10 DESC
valkey-cli --cluster call 10.0.1.11:6379 INFO memory | grep -E 'used_memory:|maxmemory:'

Interpretation: CLUSTER SLOT-STATS (since 8.0) returns key-count per slot, plus CPU and network bytes when cluster-slot-stats-enabled is yes, which finds the write-heavy slots that grow the buffer. The range's share of keys times the source's used_memory gives a rough size; keep the target well under maxmemory after adding it.

How do you start and monitor an atomic slot migration?

Get the target primary's node ID and send MIGRATESLOTS to the source. Repeat SLOTSRANGE and NODE blocks to start several jobs at once; the command returns OK only if all of them start.

TARGET=$(valkey-cli -h 10.0.1.21 CLUSTER MYID)
valkey-cli -h 10.0.1.11 CLUSTER MIGRATESLOTS SLOTSRANGE 0 511 NODE "$TARGET"

Interpretation: OK means the job started, not that it finished. The work is asynchronous, so poll next.

The CLUSTER GETSLOTMIGRATIONS reference lists one entry per job on the source primary, the target primary and the target's replicas. Terminal states are success, failed and cancelled. In the 9.1.3 source an export passes through snapshotting, replicating, waiting-to-pause, failover-paused and failover-granted, after some connection states.

# Poll the source every 5 seconds until a terminal state
while true; do
  out=$(valkey-cli -h 10.0.1.11 CLUSTER GETSLOTMIGRATIONS)
  state=$(printf '%s\n' "$out" | grep -m1 -A1 '^state$' | tail -1)
  echo "$(date +%T) $state"
  case "$state" in success|failed|cancelled) break ;; esac
  sleep 5
done
# The source log records each phase
grep -E 'Slot migration initiated|Pausing writes|now owned by target|Write loss risk' \
  /var/log/valkey/valkey.log

Interpretation: piped output puts one field per line, and the newest job comes first, so the first state match is the current job. A remaining_repl_size that keeps growing means writes outpace the link and the job will not reach the pause. Jobs live only in memory, so a restart clears them and cluster-slot-migration-log-max-len (default 1000) caps the finished ones kept; the log is your durable record.

How do you cancel a slot migration, and what happens when one fails?

The CLUSTER CANCELSLOTMIGRATIONS reference says it cancels all in-progress migrations started on that node. There is no per-job cancel, and a target cannot cancel an import.

valkey-cli -h 10.0.1.11 CLUSTER CANCELSLOTMIGRATIONS
# Every migration in the cluster
valkey-cli --cluster call 10.0.1.11:6379 CLUSTER CANCELSLOTMIGRATIONS --cluster-only-primaries

Interpretation: OK moves each export to cancelled; "No migrations ongoing" means that node was exporting nothing. The target's staged slots were never announced, so the source keeps ownership and all its data.

The 9.1.3 state machine lists the causes of an automatic failure: a lost connection, no acknowledgement for longer than repl-timeout (default 60 s), OOM on the target, a source buffer overrun, a failover or ownership change on either shard, FLUSHDB or FLUSHALL on either node, or a failed snapshot child. The message field names the cause, and the target's entry often has more. Fix it and start again; the retry begins from scratch.

Can valkey-cli reshard and rebalance with atomic slot migration?

From 9.1.0, if you ask. PR #2755 added --cluster-use-atomic-slot-migration to --cluster reshard and --cluster rebalance, as the 9.1 release post notes. Without it, both still use MIGRATE. The 9.0.7 valkey-cli has no atomic slot migration code, so on 9.0 send the commands yourself.

valkey-cli --cluster rebalance 10.0.1.11:6379 --cluster-simulate
valkey-cli --cluster rebalance 10.0.1.11:6379 --cluster-use-atomic-slot-migration

valkey-cli --cluster reshard 10.0.1.11:6379 \
  --cluster-from "$SOURCE" --cluster-to "$TARGET" --cluster-slots 512 \
  --cluster-yes --cluster-use-atomic-slot-migration

Interpretation: with the flag, valkey-cli sends one MIGRATESLOTS per source and target pair, polls every 100 ms, and prints the job's error on failure or cancellation. In 9.1.3 the flag is missing from the --cluster help option list, so the help text will not confirm it.

How does atomic slot migration interact with replicas and older versions?

The target primary forwards the migration stream to its replicas, which import alongside it. If the target shard fails over mid-migration, the import fails and you rerun the job against the new primary.

Older replicas are the trap. PR #2735 lets a 9.1 or later primary full-sync 7.2 and 8.x replicas in RDB 11, but "if any new features that the replica can't handle (such as hash-field expiration and atomic slot migration) are in use, the full sync is aborted". A node with an active import logs "Can't store slot migrations in RDB version 11". The RDB version and rollback guide covers that mixed-version window.

grep "Can't store slot migrations in RDB version" /var/log/valkey/valkey.log

Interpretation: any hit means an old replica lost its full sync during a move. Finish the version upgrade on every node before you reshard.

What changes when cluster-databases is greater than 1?

Valkey 9.0 allows numbered databases in cluster mode through cluster-databases (default 1, not changeable at runtime). The snapshot selects each database with keys in the moving slots, but PR #4952 explains that the parent never saw those SELECT commands, so a write streamed afterwards could run in the wrong database on the target. The 9.0.7 and 9.1.3 release notes list the fix.

valkey-cli -h 10.0.1.11 CONFIG GET cluster-databases
valkey-cli --cluster call 10.0.1.11:6379 INFO keyspace

Interpretation: at 1, the bug does not apply. Above 1, if you moved busy slots on 9.0.6 or 9.1.2 or older, look for keys in unexpected databases on the new owner; the fix does not repair data already moved. The migration user also needs +select. Legacy valkey-cli --cluster reshard loops over every database, although the cluster specification still says MIGRATE needs no database other than 0.

What is a safe resharding runbook with rollback?

Move slots in batches you can afford to cancel, and check the cluster between batches.

# Roll back a finished batch: the new owner is now the source
ORIGINAL=$(valkey-cli -h 10.0.1.11 CLUSTER MYID)
valkey-cli -h 10.0.1.21 CLUSTER MIGRATESLOTS SLOTSRANGE 0 511 NODE "$ORIGINAL"

Interpretation: this is another migration with the same headroom checks, so the original node must still have room. Legacy valkey-cli --cluster reshard without the flag remains a fallback, with its -ASK and large-key costs.

Frequently asked questions

Key takeaways

Plan a Valkey Cluster reshard with JusDB

JusDB's Valkey Cluster team can review your shard sizes, versions and ACLs, plan slot batches that fit your headroom, and write a resharding runbook with cancel and rollback steps for your cluster.

Bring your topology, node versions, cluster-databases setting, memory figures, and your maintenance windows. Your team approves every production change.

Related reading

• Redis Cluster: setup, resharding, and operational guide
• Valkey performance tuning: memory, persistence, and cluster config
• Valkey 9.1: database ACLs, Lua, and I/O threading
• Valkey documentation: Atomic slot migration