A Latency Incident Runbook for Valkey 9: Find the Source of a Spike

Your Valkey p99 has jumped from under a millisecond to 300 ms, the application logs are full of timeouts, and LATENCY LATEST shows an event called expire-del that the latency monitoring page never mentions. Valkey 9.1.3 is the current release (26 October 2026), but the official latency troubleshooting guide is still the Redis-era checklist. It tells you to read the "Slow Log", which Valkey 8.1 folded into COMMANDLOG. It never mentions I/O threads. Its event list names 16 events; the 9.1.3 source emits 27. This runbook covers the incident itself: confirm the delay is inside the server, find the code path blocking the main thread, and capture evidence before anything is reset. For configuration work outside an incident, read our general Valkey performance tuning guide instead.

Scope and review date: November 2026. This runbook covers self-managed Valkey 9.1, where 9.1.3 is the current patch, and notes where 9.0 or 8.1 differ and where the 9.2 release candidate changes behavior. Event names, defaults, and output formats were checked against the Valkey documentation, release notes, and 9.1.3 source code on 10 November 2026. The commands were not run in a lab and the terminal output is illustrative, so rehearse on staging. Managed services may block some commands.

Is the latency coming from Valkey, the network, or the client?

The latency a client sees is the sum of time in its own connection pool, the network round trip, time in Valkey's I/O path, and command execution on the main thread. Valkey's own instruments see only the last part. The command latency event and the slow COMMANDLOG both time execution and exclude reading the request and writing the reply, as the COMMANDLOG GET reference states. So start by measuring the round trip from both ends.

# 1. From an application host
valkey-cli -h valkey-01 -p 6379 --latency-history -i 5

# 2. From the Valkey host itself, against the same instance
valkey-cli -p 6379 --latency-history -i 5

Interpretation: both runs time PING round trips and print min, max, and average per interval (default 15 seconds). If the application host shows spikes and the local run stays flat, look at the network, TLS, or the client. If both spike together, the server is the bottleneck.

valkey-cli INFO latencystats
valkey-cli INFO stats | grep -E 'total_connections_received|rejected_connections'

Interpretation: latency_percentiles_usec_<command> lines give p50, p99, and p99.9 of execution time per command since startup or the last CONFIG RESETSTAT. A flat server-side p99 while the client p99 climbs means the time is spent outside execution. A fast-rising total_connections_received points to a client reconnect storm.

On the client, check pool exhaustion, retry storms, and garbage-collection pauses. Valkey 9.0 turned on TCP_NODELAY by default for its own sockets (9.0.0-rc1 release notes, #1763); the client library sets its own.

What should you run first when Valkey latency spikes?

Work from the outside in, and read before you change anything. Only step 3 changes a runtime setting.

How do you read LATENCY LATEST, HISTORY, and DOCTOR in Valkey 9.1?

The latency monitor is off by default. Per the latency monitoring documentation, you enable it by setting latency-monitor-threshold in milliseconds. Each event at or above it is recorded in a 160-sample time series, with spikes in the same second merged by keeping the maximum. At 0 nothing is recorded, so an unconfigured instance has no history of the incident you are in.

valkey-cli CONFIG SET latency-monitor-threshold 10
valkey-cli LATENCY LATEST
valkey-cli LATENCY HISTORY expire-del

Interpretation: since Valkey 8.1, each LATENCY LATEST entry has six fields: the event name, the Unix time of the latest spike, its latency in milliseconds, the all-time maximum, and the sum and count of recorded spikes (LATENCY LATEST; the page's Reply section still says four elements). LATENCY HISTORY returns the time and latency pairs for one event, so you can line spikes up with deploys, cron jobs, and BGSAVE times.

Choose a threshold below your latency budget. 10 ms matches the default slow COMMANDLOG threshold, and in the 9.1.3 source LATENCY DOCTOR warns when the slow log threshold is the higher of the two. Put latency-monitor-threshold 10 in valkey.conf so it survives restarts.

LATENCY DOCTOR turns the same data into a report with advice, and in 9.1.3 the advice lags the server. It says to check the "Slow Log" with slowlog-log-slower-than. It says to disable transparent huge pages with echo never, while the startup check recommends madvise. And it has advice only for the documented events, so newer events get statistics but no hint. All three are visible in the 9.1.3 latency.c source.

Which latency events does Valkey 9.1 emit that the docs do not list?

A search of the 9.1.3 source for latency samples finds 27 event names. The documentation lists 16. Fifteen match; aof-rewrite-diff-write is never emitted. The 9.2.0-rc2 source has the same 27, and modules can add their own.

What the undocumented events time, from the 9.1.3 source:

How do you find slow and oversized commands with COMMANDLOG?

Valkey 8.1 added COMMANDLOG (#1294 in the 8.1.0 release notes). It keeps three logs: slow for execution time, large-request for request size, and large-reply for reply size. In the 9.1.3 valkey.conf, commandlog-execution-slower-than defaults to 10000 microseconds, commandlog-request-larger-than and commandlog-reply-larger-than to 1048576 bytes, and each log keeps 128 entries.

A threshold of -1 disables a log and 0 logs every command. The old slowlog-log-slower-than and slowlog-max-len settings are aliases for the slow-type settings, and valkey.conf calls them deprecated. SLOWLOG GET still works and reads the same slow log, so existing dashboards keep working.

valkey-cli COMMANDLOG GET 20 slow
valkey-cli COMMANDLOG GET 20 large-request
valkey-cli COMMANDLOG GET 20 large-reply

Interpretation: unlike SLOWLOG GET, which defaults to 10 entries, COMMANDLOG GET needs both a count and a type; -1 returns every entry. Each entry has six values: an ID, a Unix timestamp, the measured value (microseconds for slow, bytes for the other two), the command and its arguments, the client address, and the client name.

How do I/O threads change what you see during a spike?

With io-threads above 1, I/O threads read and parse requests, write replies, and poll sockets. Command execution stays on the main thread, as the latency monitoring page says. The count includes the main thread, so io-threads 1 means no extra threads. Since 9.0 it can be changed at runtime (#2033); avoid doing that mid-incident. Our Valkey 9.1 upgrade guide covers what changed in 9.1 threading.

Two consequences follow. Time a request waits for an I/O thread, or spends writing a large reply, is invisible to the latency monitor and the slow log. And CPU graphs mislead: the INFO reference explains that Valkey busy-waits when I/O threads are on, so CPU time looks close to 100% even with spare capacity. Valkey 9.1 added active-time counters for this.

valkey-cli INFO cpu | grep used_active_time; sleep 10
valkey-cli INFO cpu | grep used_active_time

Interpretation: each field is seconds of real work since startup; subtract the samples and divide by 10. A main thread active for 9.5 of 10 seconds is saturated, and more I/O threads will not help. Busy I/O threads with an idle main thread point to network work, such as large replies or TLS.

Check the patch level too. 9.1.2 fixed a client deadlock with BLPOP followed by a partly delivered pipeline (#4531) and a frozen monotonic clock on hosts with unsynchronized TSC that stopped key expiration (#4346). 9.1.3 fixed event loop busy-spinning on VMs whose kernel clocksource is not tsc (#4707, 9.1.3 release notes). On a VM running 9.1.0 to 9.1.2, compare monotonic_clock in INFO server with the kernel clocksource; the fix is the upgrade.

Are big keys or blocking commands stalling the event loop?

One command on a large value blocks every other client for as long as it runs. The usual culprits are KEYS, and full reads of large collections such as HGETALL, SMEMBERS, or ZRANGE key 0 -1. Set operations like SUNION and SORT on large inputs are also common. They show up as the command event, in the slow log, and often in the large-reply log. BLPOP and similar commands only park one client.

# Run on a replica: these scan the keyspace
valkey-cli -h valkey-replica-01 --bigkeys -i 0.01
valkey-cli -h valkey-replica-01 --memkeys -i 0.01
valkey-cli MEMORY USAGE session:index SAMPLES 0

Interpretation: --bigkeys reports the largest key per type by element count, and --memkeys by memory. -i 0.01 sleeps 10 ms every 100 scan cycles to reduce load. SAMPLES 0 measures every element, which is itself slow on a huge key.

Deleting big keys is less dangerous than it was. Since Valkey 8.0, lazy freeing is the default: lazyfree-lazy-eviction, lazyfree-lazy-expire, lazyfree-lazy-server-del, lazyfree-lazy-user-del, lazyfree-lazy-user-flush, and replica-lazy-flush are all yes in valkey.conf. Configurations carried over from older versions sometimes set them to no.

valkey-cli CONFIG GET 'lazyfree-*'
valkey-cli CONFIG GET busy-reply-threshold

Interpretation: any no in the first result means deletes of that kind free memory on the main thread. busy-reply-threshold (alias lua-time-limit, default 5000 ms) is how long a script or function runs before other clients get a BUSY error. A while-blocked-cron event during the spike confirms the server was stuck in a long script, a module call, or a load.

Are expiry or eviction cycles causing the spikes?

Valkey removes expired keys when they are accessed and in an active expire cycle. valkey.conf describes the default active-expire-effort 1 (range 1 to 10) as keeping expired-but-present keys under about 10%. In the 9.1.3 source, at the default effort, the slow cycle may use up to 25% of CPU time and a fast cycle is capped at 1,000 microseconds. Since 9.0, the same cycle also reclaims expired hash fields. When many keys share one expiry time, such as a batch load with a fixed TTL, you see expire-cycle spikes.

valkey-cli INFO stats | grep -E 'expired_|evicted_|eviction_exceeded'
valkey-cli INFO memory | grep -E '^used_memory:|^maxmemory:|^maxmemory_policy:'

Interpretation: a rising expired_time_cap_reached_count means active expiry cycles are stopping early on their time limit, and a high expired_stale_perc means expired keys are piling up. Spread TTLs with random jitter instead of raising the effort. On the eviction side, total_eviction_exceeded_time and current_eviction_exceeded_time (milliseconds) show how long used_memory has been above maxmemory.

Eviction runs before commands that can add memory whenever usage is over the limit, and appears as eviction-cycle and eviction-del. Its time budget comes from maxmemory-eviction-tenacity (default 10; valkey.conf describes 0 as minimum latency and 100 as no regard for latency). Lowering maxmemory suddenly forces mass eviction. For expiry spikes, LATENCY DOCTOR suggests hz 100 (default 10), which makes cycles smaller but more frequent, so test it first.

Is the host adding latency through fork, huge pages, or swap?

BGSAVE, AOF rewrite, and full resynchronization of a replica all fork. The fork() call runs on the main thread and copies the page table. The latency troubleshooting guide gives the example of a 24 GB instance needing a 48 MB page table. After the fork, each page the parent writes is copied (copy-on-write).

valkey-cli INFO stats | grep latest_fork_usec
valkey-cli INFO persistence | grep -E 'bgsave_in_progress|rewrite_in_progress|_cow_size'

Interpretation: latest_fork_usec is the last fork's duration in microseconds, and the fork event records forks above the threshold. Spikes that line up with rdb_bgsave_in_progress:1 point to persistence. LATENCY DOCTOR also rates the fork rate in GB per second: under 10 is "terrible" and under 25 is "poor".

Transparent huge pages (THP) make copy-on-write far more expensive. Valkey ships disable-thp yes, which turns THP off for the Valkey process when the kernel is set to always, and the 9.1.3 startup check recommends madvise. The documentation and the 9.1.3 LATENCY DOCTOR still say echo never; the 9.2 release candidate changes the doctor to recommend madvise (#3947). Either value stops THP from applying to Valkey by default.

cat /sys/kernel/mm/transparent_hugepage/enabled
pid=$(valkey-cli INFO server | awk -F: '/^process_id/{print $2}' | tr -d '\r')
grep -E '^(Swap|AnonHugePages):' /proc/$pid/smaps \
  | awk '{t[$1]+=$2} END {for (k in t) print k, t[k], "kB"}'
vmstat 1 5

Interpretation: non-zero AnonHugePages means the process is still using huge pages, and LATENCY DOCTOR notes that a restart is needed to drop pages that already exist. More than a few kB of Swap, or non-zero si and so columns in vmstat, means pages are moving to and from disk. The INFO reference adds that used_memory well above used_memory_rss means part of Valkey's memory has been swapped out. Then find the pressure behind it: other processes, fork copy-on-write growth, or a replica past maxmemory (9.1.3 fixed one cause, #4920).

What should you capture for the postmortem?

Most evidence lives in memory, and a restart, failover, LATENCY RESET, COMMANDLOG RESET, or CONFIG RESETSTAT erases it. Capture first, then fix. Save host data such as vmstat output alongside.

#!/usr/bin/env bash
# Run against the affected node; add -h/-p/--user as needed.
out="valkey-incident-$(date -u +%Y%m%dT%H%M%SZ)"; mkdir -p "$out"
cli() { valkey-cli "$@"; }
cli INFO everything            > "$out/info.txt"
cli INFO debug                 > "$out/info-debug.txt"
cli LATENCY LATEST             > "$out/latency-latest.txt"
cli LATENCY DOCTOR             > "$out/latency-doctor.txt"
for e in $(cli --raw LATENCY LATEST | awk 'NR % 6 == 1'); do
  cli LATENCY HISTORY "$e"     > "$out/latency-history-$e.txt"
done
for t in slow large-request large-reply; do
  cli COMMANDLOG GET -1 "$t"   > "$out/commandlog-$t.txt"
done
cli CONFIG GET '*'             > "$out/config.txt"

Interpretation: INFO everything includes the latencystats and module sections. INFO debug holds eventloop_duration_max and related event-loop counters, which are not in the default output. The history loop reads every sixth raw line as an event name, so check its files. Command logs hold key names and client addresses, so store the folder like production data.

In the write-up, record the version, the event names and worst values, the top command log entries, fork times, and the threshold in force, lined up against client-side graphs. If the server shows nothing during a client-visible spike, say so: that moves the investigation to the network or the client.

Frequently asked questions

Key takeaways

Get help with Valkey latency with JusDB

JusDB's Valkey performance tuning team can review your latency monitor and command log data after an incident, check I/O threads, persistence, and host settings against your workload, and help you write a latency runbook for your own topology.

Bring the capture folder from the last section, your Valkey version, and your client-side latency graphs. Your team approves every production change.

Related reading

• Valkey performance tuning: memory, persistence, and cluster config
• Valkey 9.1 for operators: what changed and how to upgrade
• Redis persistence: RDB vs AOF, which to choose
• Valkey documentation: Latency monitoring