Sentinel on Valkey 9: Coordinated Failover and the ACL Change That Broke Failovers

You upgraded your Sentinels to Valkey 9.0.0, a primary went down, and nothing happened for three minutes. Then the Sentinel log printed -failover-abort-slave-timeout and the replica was still a replica. The cause was not the network or the quorum. Valkey 9.0.0 Sentinel started adding FAILOVER ABORT to every reconfiguration it sends, so a dedicated Sentinel user with the least-privilege ACL documented before 9.0 could no longer promote anything. Valkey 9.0.1 fixed it on 9 December 2025, but the documentation now asks for a broader ACL, and Valkey 9.0 also added coordinated failover, which does need new permissions. This post covers how to detect the 9.0.0 regression, the Sentinel ACL that works on 9.x, what SENTINEL FAILOVER ... COORDINATED does, the settings and commands renamed since 8.0, and an upgrade runbook from 7.2 or 8.x to 9.1.3.

Scope and review date: November 2026. This post covers self-managed Valkey Sentinel deployments on Valkey 7.2, 8.0, 8.1, 9.0, and 9.1, with 9.1.3 (26 October 2026) and 9.0.7 (27 October 2026) as the current patches. Commands, defaults, and behavior were checked against the Valkey documentation, the GitHub release notes, and the Sentinel source at the 9.0.0, 9.0.1, and 9.1.3 tags. They were not run in a lab, so rehearse them on staging.

If you are still deciding whether Sentinel is the right topology, read Valkey Cluster vs Sentinel: choosing an HA topology first. This post assumes Sentinel and covers running it on 9.x.

What broke Sentinel failovers in Valkey 9.0.0?

Sentinel never sends a bare REPLICAOF NO ONE. It wraps every role change in a transaction: MULTI, SLAVEOF, CONFIG REWRITE, two CLIENT KILL TYPE commands, then EXEC. The same function promotes the chosen replica, points the other replicas at the new primary, and demotes an old primary that comes back. Valkey 9.0 added FAILOVER ABORT as the first queued command, so Sentinel can recover an instance stuck in a FAILOVER command.

In 9.0.0 the condition was too broad. Sentinel added FAILOVER ABORT for any instance that reports master_failover_state in INFO replication, and every Valkey version reports that field. When the Sentinel user lacks +failover, the server rejects the queued command with NOPERM and marks the transaction failed. EXEC then returns EXECABORT, so SLAVEOF NO ONE never runs. Sentinel ignores the replies and waits for INFO to show the new role. After failover-timeout (default 180000 ms, 3 minutes) it logs -failover-abort-slave-timeout. The shipped sentinel.conf says the same Sentinel will not retry against that primary for twice the failover timeout.

The bug was reported as issue #2779 on 27 October 2025, six days after 9.0.0 shipped. PR #2780 merged on 31 October 2025 and shipped in Valkey 9.0.1 on 9 December 2025. The fix is one condition: send FAILOVER ABORT only when the instance reports a failover in progress.

# The whole fix in src/sentinel.c, 9.0.0 to 9.0.1
git diff 9.0.0 9.0.1 -- src/sentinel.c
-    if (ri->monitored_instance_failover_state != SENTINEL_MONITORED_INSTANCE_FAILOVER_NS) {
+    if (ri->monitored_instance_failover_state == SENTINEL_MONITORED_INSTANCE_FAILOVER) {

Interpretation: FAILOVER_NS means "the instance does not support FAILOVER". Every Valkey instance supports it, so 9.0.0 sent the abort every time. From 9.0.1 the abort is sent only when master_failover_state is not no-failover, which is the stuck-failover case it was written for.

How do you tell whether your Sentinels hit the 9.0.0 ACL regression?

Three conditions must all hold: a Sentinel runs 9.0.0, it authenticates to the data nodes as a dedicated user (sentinel auth-user), and that user lacks +failover. The data nodes' version does not matter, because Valkey 7.2 and 8.x also report master_failover_state. A Sentinel that connects as the default user with all permissions was not affected.

# 1. Sentinel version: run against every Sentinel
valkey-cli -p 26379 INFO server | grep -E '^valkey_version'

# 2. Can the Sentinel user run FAILOVER ABORT? Run on every data node as an admin user
valkey-cli -p 6379 --user admin --askpass ACL DRYRUN sentinel-user FAILOVER ABORT

Interpretation: valkey_version:9.0.0 on any Sentinel is the first condition. ACL DRYRUN replies OK when the user may run the command, and otherwise returns a string such as User sentinel-user has no permissions to run the 'failover' command. If both checks fail, your next failover will abort.

If a failover has already failed, the evidence is on the replica Sentinel tried to promote. Each rejected command adds an entry to its ACL log with the context multi, because it was rejected while being queued.

Fix it in this order. First add +failover to the Sentinel user on every data node, which makes 9.0.0 Sentinels work immediately. Then upgrade the Sentinels to 9.0.7 or 9.1.3. Adding the permission first means a primary failure during the upgrade window still fails over.

Which ACL does the Sentinel user need on Valkey 9.x?

The least-privilege ACL that the Valkey documentation showed before 9.0 is the one issue #2779 quotes, with explicit subcommands and no FAILOVER:

# Pre-9.0 documented ACL, as quoted in issue #2779
ACL SETUSER sentinel-user ON >somepassword allchannels +multi +slaveof +ping +exec \
  +subscribe +config|rewrite +role +publish +info +client|setname +client|kill +script|kill

Interpretation: this still works on 9.0.1 and later for automatic failover and for plain SENTINEL FAILOVER. It does not work for coordinated failover, which needs FAILOVER plus CLIENT PAUSE and CLIENT UNPAUSE. Note that the grant is +slaveof, not +replicaof: Sentinel sends SLAVEOF, and the two are separate commands for ACL purposes.

The Sentinel documentation now says "Since Valkey Sentinel 9.0, the sentinel user requires the +failover and +client permissions for proper operation." That wording was written after the fix, and as PR #2780 states, only coordinated failover strictly needs the new permissions on 9.0.1 and later. Granting them anyway is still the right default: it covers coordinated failover and lets Sentinel abort a stuck FAILOVER.

We derived the full list of commands Sentinel sends to data nodes from src/sentinel.c at 9.1.3: PING, INFO, CLIENT SETNAME, SUBSCRIBE and PUBLISH on __sentinel__:hello, MULTI, EXEC, SLAVEOF, CONFIG REWRITE, CLIENT KILL, SCRIPT KILL, and, for 9.x, FAILOVER, CLIENT PAUSE, and CLIENT UNPAUSE. A subcommand-scoped user for 9.x looks like this:

# Run on every primary and replica, then persist it
ACL SETUSER sentinel-user ON >somepassword resetchannels &__sentinel__:hello \
  +ping +info +multi +exec +slaveof +config|rewrite +script|kill \
  +client|setname +client|kill +client|pause +client|unpause +failover \
  +subscribe +publish

ACL SAVE        # if users live in an aclfile
CONFIG REWRITE  # if users are defined in valkey.conf

Interpretation: this is narrower than the documentation's example, which grants all of +client and +config. If you prefer the documented form, use +client +config in place of the subcommands. Either way, run one automatic and one coordinated failover on staging and confirm ACL LOG stays empty on every node.

Two related settings sit on the Sentinel side. sentinel auth-user and sentinel auth-pass tell each Sentinel which user to authenticate as. If you renamed FAILOVER on the data nodes, Sentinel also needs a rename-command entry, or it sends a command the server does not know.

# sentinel.conf, on every Sentinel
sentinel auth-user mymaster sentinel-user
sentinel auth-pass mymaster somepassword
# Only if FAILOVER is renamed on the data nodes:
# sentinel rename-command mymaster FAILOVER FAILOVER-7f3a

Interpretation: rename-command maps every command Sentinel sends, including the FAILOVER that 9.x added. A rename that predates 9.0 has no FAILOVER entry, so check it during the upgrade.

What does SENTINEL FAILOVER COORDINATED do differently?

A classic Sentinel failover promotes a replica with SLAVEOF NO ONE while the old primary is unreachable or still accepting writes. Writes the old primary acknowledged but had not replicated are lost, because Valkey replication is asynchronous. That is acceptable for a crash, but a planned switchover can do better.

SENTINEL FAILOVER <name> COORDINATED, added in Valkey 9.0 by PR #1292, hands the role over with the server's own FAILOVER command. From the 9.1.3 source, the sequence is:

1. The Sentinel runs a leader election among the Sentinels. Plain SENTINEL FAILOVER skips the election; the coordinated form needs a majority and the quorum, like an automatic failover.
2. It sends the primary MULTI, CLIENT PAUSE <ms> WRITE, FAILOVER TO <replica> <port> TIMEOUT <ms>, EXEC, where the timeout is what remains of failover-timeout.
3. The primary waits for the replica to reach its replication offset, demotes itself, and sends PSYNC FAILOVER so the replica promotes itself.
4. When INFO shows the promotion, Sentinel runs CONFIG REWRITE, kills normal and pub/sub client connections, and runs CLIENT UNPAUSE on both nodes. Clients reconnect and ask Sentinel for the new primary.
5. Sentinel reconfigures the remaining replicas with SLAVEOF, as in a classic failover.

# On one Sentinel
valkey-cli -p 26379 SENTINEL FAILOVER mymaster COORDINATED

# On the current primary, while it runs
valkey-cli -p 6379 --user admin --askpass INFO replication | grep -E '^(role|master_failover_state)'

Interpretation: master_failover_state moves from no-failover to waiting-for-sync to failover-in-progress, then the node reports role:slave. Sentinel does not pass FORCE, so if the replica does not catch up before the timeout, the primary rolls back and stays primary. The reply NOGOODPRIMARY Primary does not support FAILOVER command means Sentinel has not yet read the primary's failover state from INFO.

Coordinated failover is for planned work on a healthy primary. It needs a working connection to the primary, so it cannot help when the primary has crashed; Sentinel keeps retrying until failover-timeout and then logs -failover-abort-master-timeout or -failover-abort-slave-timeout.

What do clients need to handle coordinated failover?

The Sentinel documentation warns that "clients that rely only on Sentinel pub/sub messages risk reconnecting to the old primary after it has been demoted to a replica, which can result in READONLY errors". The documentation notes that coordinated failovers "can complete very quickly", so a client that reconnects to a cached address on +switch-master can land on the demoted node.

The Sentinel client guidelines describe the safe pattern: ask a Sentinel for the primary address, connect, and confirm the role before sending writes. Treat pub/sub events as a hint to repeat that check.

# What a correct client does after any disconnect
valkey-cli -p 26379 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster
valkey-cli -h 10.0.0.11 -p 6379 --user app --askpass ROLE

Interpretation: the first reply is the address Sentinel believes is primary. ROLE must return master as its first element; if it returns slave, the client asks Sentinel again. Test your client library with a coordinated failover on staging before using it in production; a library that passes classic failover drills can still fail this one.

Which Sentinel settings and commands were renamed since 8.0?

Valkey 8.0 renamed one Sentinel setting and added four inclusive command names. None of the old names were removed in 9.1.3, but the new names do not exist on 7.2, so mixed fleets need care.

The sources are the 8.0.0-rc1 and 8.0.0-rc2 release notes. The Sentinel documentation page lists neither the setting nor the aliases.

# The aliases are separate ACL subcommands on the Sentinel itself.
# If clients switch to the new names, grant both on every Sentinel:
ACL SETUSER sentinel-client +sentinel|get-master-addr-by-name +sentinel|get-primary-addr-by-name

Interpretation: in 9.1.3 the aliases have their own command definitions (GET-PRIMARY-ADDR-BY-NAME is marked "since 8.0.0"), so a client ACL that allows only the old name rejects the new one. Keep clients on the old names until every Sentinel runs 8.0 or later.

primary-reboot-down-after-period and empty primaries

The replication documentation explains why this setting matters. Sentinel identifies instances by address, so a primary that restarts with no persistence, or a replacement that reuses its address, is taken for the same primary, and its replicas copy the empty dataset. Sentinel logs a +reboot event but draws no conclusion from it by default.

# On every Sentinel; the value is in milliseconds, default 0 (off)
valkey-cli -p 26379 SENTINEL SET mymaster primary-reboot-down-after-period 30000

Interpretation: with a non-zero value, Sentinel treats a rebooted primary that keeps answering -LOADING for longer than this period as down and fails over. As the replication page notes, it forces a failover only while the primary stays unresponsive, so it does not protect against a fast empty restart. The real fix is persistence or no automatic restart for a primary under Sentinel.

One rollback trap: once a Sentinel rewrites its config with a non-zero value, the file contains primary-reboot-down-after-period. A Valkey 7.2 Sentinel knows only the old name and exits at startup with "Unrecognized sentinel configuration statement".

How do you upgrade Sentinel from 7.2 or 8.x to 9.1?

The Valkey administration guide gives the data-node pattern: upgrade replicas one at a time, fail over to an upgraded replica, then upgrade the old primary. For Sentinel fleets we add two things: fix the ACL before any Sentinel runs 9.x, and upgrade the Sentinels first so the failover you use for the data nodes is driven by fixed code. Target 9.1.3, or 9.0.7 if you stay on 9.0.

# Before and after every Sentinel change
valkey-cli -p 26379 SENTINEL CKQUORUM mymaster
valkey-cli -p 26379 INFO sentinel | grep -E '^sentinel_(tilt|total_tilt)'

Interpretation: CKQUORUM should return OK 3 usable Sentinels. Quorum and failover authorization can be reached for a three-Sentinel fleet. sentinel_total_tilt, added in 9.0, counts how often this Sentinel entered TILT mode; earlier versions do not report it.

Two details are easy to miss. Upgrade the Sentinels before the data nodes, because a 9.1.3 Sentinel sends only commands that 7.2 and 8.x servers also support, and the ACL from step 2 is already in place. And if you upgrade data nodes from 8.x, the RDB format changes: Valkey 9.0 and 9.1 write RDB version 80, which 8.x cannot load. Our Valkey 9.1 upgrade guide covers the data-node rollback limits in detail.

Frequently asked questions

Key takeaways

Run Valkey Sentinel on 9.x with JusDB

JusDB's Valkey high availability team can review your Sentinel topology, quorum, and ACLs against the 9.x requirements, plan the Sentinel and data-node upgrade order, and rehearse coordinated failover with your client libraries on staging.

Bring your Sentinel and data-node versions, your sentinel.conf and ACL files, and the client libraries your services use. Your team approves every production change.

Related reading

• Redis Sentinel vs Redis Cluster: choosing the right HA architecture
• Valkey 9.1 for operators: what changed and how to upgrade
• Redis 8 vs Valkey 8 in production
• Valkey documentation: high availability with Valkey Sentinel