Your small-key traffic is healthy until a batch job fetches a few 10 MB blobs, and then p99.9 for every client on the instance jumps from under a millisecond to tens of milliseconds. Nothing shows in the slow log, because each GET executes quickly. The time goes into building and writing the reply. Valkey 9.0 added reply copy avoidance, which stops copying large string values into the reply buffer, and Valkey 8.1 added the large-request and large-reply types to COMMANDLOG so you can see oversized traffic. The two features interact: the 9.0.1 fix that made large replies visible to COMMANDLOG again cost throughput, and 9.0.2 added a way to get it back. This guide explains the mechanism, the settings, and how to find and fix the values behind the spikes.
- One multi-MB reply delays everyone because the main thread runs every command and the write loop serves each client in turns of a little over 64 KB.
- Valkey 9.0 reply copy avoidance writes a reference to the stored string instead of copying it. In 9.1.3 source it applies to raw strings of 16 KB or more without I/O threads, 64 KB or more with 2 to 6
io-threads, and any size at 7 or more. COMMANDLOG GET 10 large-replylists replies abovecommandlog-reply-larger-than(default 1048576 bytes). On 9.0.2 and later, setting it to-1removes the byte tracking cost that 9.0.1 introduced.- Find big values with
valkey-cli --memkeysandMEMORY USAGEon a replica, then chunk them, compress them in the client, or move them to object storage.
Scope and review date: November 2026. This post covers self-managed Valkey 9.0 and 9.1 (current patches 9.0.7 and 9.1.3, released 27 and 26 October 2026), with notes on 8.1 and the 9.2 release candidates. Config names, defaults, and thresholds were checked against the Valkey documentation, the GitHub release notes, and the 9.1.3 and 9.0.0 source on 10 November 2026. The commands were not run in a lab, and the only latency figures quoted are the vendors' own.
Why does one multi-MB value raise p99.9 for every client?
Valkey executes commands on one main thread. A GET of a 10 MB string is fast to look up, but its reply is large, and three costs follow it.
- Reply construction. Before 9.0, the main thread copied the whole value into the client's reply buffer with
memcpy. A 10 MB value meant 10 MB of copying on the thread that serves everyone else. - Write turns. The write loop stops after a little over 64 KB per client per pass (
NET_MAX_WRITES_PER_EVENTinserver.h) so that one reply cannot hold the loop. The rest goes out on later passes, and small replies queue behind each pass. The loop ignores this limit when used memory is abovemaxmemory. - The network. 10 MiB is about 84 Mbit, so on a 10 Gbit/s link it needs about 8.4 ms of wire time before protocol overhead (arithmetic, not a measurement). Small replies on the same connection, and on a shared NIC queue, wait behind it.
The write side has its own costs. A 10 MB SET fills the query buffer, is parsed on the main thread or an I/O thread, and is then propagated to every replica and to the AOF. The defaults that bound these sizes are generous, so large values pass through unless you lower them.
valkey-cli CONFIG GET proto-max-bulk-len
valkey-cli CONFIG GET client-query-buffer-limit
valkey-cli CONFIG GET client-output-buffer-limitInterpretation: the 9.1.3 defaults are proto-max-bulk-len 512 MB (the largest single string a client may send), client-query-buffer-limit 1 GB, and client-output-buffer-limit normal 0 0 0, which means no output limit for normal clients. Replica clients default to 256mb 64mb 60, so a burst of large writes can disconnect a replica and force a full resync.
The reply path of a large GET in Valkey 9.0 and later
Copy avoidance changes step 2: the reply buffer holds a reference, not the value
What it does not remove: the kernel still copies the bytes into the socket, and the wire time is the same. Copy avoidance takes the memcpy off the main thread, not the size off the network.
What does reply copy avoidance in Valkey 9.0 change?
Reply copy avoidance arrived in Valkey 9.0.0-rc1 as "directly writing responses to clients" (PR #2078, titled "Reply Copy Avoidance"). Valkey 9.0.0 went GA on 21 October 2025. For a string reply that qualifies, the main thread appends a small reference (a pointer to the object and a pointer to its string) instead of the bytes. The writer builds an iovec that points at the stored value and sends it with writev. With I/O threads on, that write work also leaves the main thread.
The reference holds the object alive until the write completes. If a slow client is still reading when the key is overwritten or deleted, the old value's memory is freed only after that client has received it.
The main published measurement is a Valkey blog post by Momento (3 November 2026). It ran 100K requests per second of 1 KB GETs over 256 connections, each pipelining 32 requests, with 10 requests per second of 10 MB GETs as background noise. These are Momento's figures; the post does not state the instance type or the io-threads count.
| 1 KB GET latency (Momento's test) | p50 | p99 | p99.9 | max |
|---|---|---|---|---|
| Valkey 8.1, no noise | 295 µs | 416 µs | 489 µs | 2.80 ms |
| Valkey 8.1, 10 MB noise | 289 µs | 500 µs | 26.2 ms | 37.2 ms |
| Valkey 9.0, no noise | 291 µs | 403 µs | 479 µs | 3.26 ms |
| Valkey 9.0, 10 MB noise | 295 µs | 799 µs | 3.10 ms | 11.9 ms |
The pattern matters more than the numbers: p50 does not move, and the damage sits at p99.9 and above. Under noise, 9.0's p99 is higher than 8.1's while its p99.9 is much lower, so a p99-only dashboard can show the upgrade as a regression. The Valkey 9.0 announcement separately says large requests avoid internal memory copying for "up to 20% higher throughput", with no conditions stated.
To reproduce the shape on staging, load one large value and run two benchmarks at once.
# One 10 MiB value (valkey-cli -x reads the last argument from stdin)
head -c 10485760 /dev/urandom | valkey-cli -x SET blob:1
# Small keys first, then the measured run (9.1 has --duration)
valkey-benchmark -t set -d 1024 -r 100000 -n 1000000 -q
valkey-benchmark -t get -d 1024 -r 100000 -c 256 -P 32 \
--rps 100000 --duration 60 --threads 4
# In a second shell, the noise: 10 large GETs per second
valkey-benchmark -c 2 --rps 10 --duration 60 GET blob:1Interpretation: read the "Latency by percentile distribution" block of the first benchmark, not the summary line, because the summary stops at p99. Compare the 99.900% row with and without the noise run. --rps arrived in 9.0 and --duration in 9.1, so use -n on older clients.
When does copy avoidance apply, and does it need io-threads?
It does not need I/O threads, but the threshold depends on them. No documentation page describes the rule. It lives in isCopyAvoidPreferred() in src/networking.c and in three hidden configs in src/config.c. The defaults are the same in the 9.0.0, 9.0.7, 9.1.3, and 9.2.0-rc2 source.
| io-threads | Copy avoidance applies to raw strings of | Hidden config (default) |
|---|---|---|
| 1 (main thread only) | 16 KB or more | min-string-size-avoid-copy-reply (16384) |
| 2 to 6 | 64 KB or more | min-string-size-avoid-copy-reply-threaded (65536) |
| 7 or more | Any size | min-io-threads-avoid-copy-reply (7) |
Other conditions apply on top. The client must be a normal or Pub/Sub client, so replicas and internal (fake) clients such as the one scripts run on are excluded. The value must be raw-encoded, so integers and short embedded strings are copied as before. The commands that take this path are the ones that reply with a whole string object, mainly GET, GETEX, GETDEL, GETSET, SET ... GET, and MGET (a single-element LPOP or LMOVE can too). HGET, HMGET, and LRANGE build their replies from a copy, so a large hash field still goes through memcpy.
# Hidden configs answer only an exact name, never a pattern such as *
valkey-cli CONFIG GET io-threads
valkey-cli CONFIG GET min-io-threads-avoid-copy-reply
valkey-cli CONFIG GET min-string-size-avoid-copy-reply
valkey-cli CONFIG GET min-string-size-avoid-copy-reply-threadedInterpretation: io-threads counts the main thread, so 1 means no I/O threads. If the three hidden values match the defaults above, the table applies to your build. Read them, but leave them alone: they are undocumented, can change between releases, and managed services may block CONFIG.
Copy avoidance helps most when the write runs on an I/O thread, because then neither the copy nor the writev runs on the main thread. If large values are part of normal traffic, size io-threads with measurement first; the sibling guide on sizing Valkey io-threads covers the method.
How do COMMANDLOG large-request and large-reply work?
COMMANDLOG arrived in Valkey 8.1.0 and keeps three logs. slow replaces the slow log and records execution time. large-request records commands whose arguments together exceed a size, and large-reply records commands whose reply exceeds a size. The 9.1.3 valkey.conf sets these defaults:
commandlog-execution-slower-than 10000 # microseconds
commandlog-slow-execution-max-len 128
commandlog-request-larger-than 1048576 # bytes
commandlog-large-request-max-len 128
commandlog-reply-larger-than 1048576 # bytes
commandlog-large-reply-max-len 128Interpretation: by default, any request or reply over 1 MiB is logged, and each log keeps its newest 128 entries. A negative threshold disables a log and 0 logs every command. All six settings can be changed at runtime with CONFIG SET.
valkey-cli COMMANDLOG GET 10 large-reply
valkey-cli COMMANDLOG GET 10 large-request
valkey-cli COMMANDLOG LEN large-reply
valkey-cli COMMANDLOG RESET large-replyInterpretation: each entry has six fields: a progressive ID, a Unix timestamp, the size in bytes, the command arguments, the client address, and the client name. The server keeps at most 32 arguments per entry and the first 128 bytes of each argument, so a 10 MB SET appears with its key and a truncated value. For large-reply, a GET of a 10,485,760-byte value logs 10,485,773 bytes, because the size includes the RESP header and trailing CRLF.
Reading the large-reply log and scanning for big values
Illustrative output, shaped like valkey-cli 9.1; your IDs, keys, and sizes will differ
Read field 3: it is the reply size in bytes, and field 6 names the client if your application sets CLIENT SETNAME. --memkeys reports bytes from MEMORY USAGE, so it is a little larger than the string length.
What did the 9.0.1 large-reply fix cost, and how do you get throughput back?
In 9.0.0, copy-avoided replies were not counted, so large-reply missed them. PR #2652 says replies larger than 16384 bytes were not being recorded, which is the threshold of the hidden min-string-size-avoid-copy-reply config. Valkey 9.0.1 (9 December 2025) fixed it by counting reply bytes on the copy-avoidance path for every reply. That counting cost throughput.
Valkey 9.0.2 (3 February 2026) made the counting conditional: "Track reply bytes in I/O threads if commandlog-reply-larger-than is -1 (#3086, #3126). This makes it possible to mitigate a performance regression in 9.0.1 caused by the bug fix #2652." The same change is in 9.1.0-rc1, and the 9.1.3 source checks the threshold before counting. The 9.1.3 valkey.conf now warns that any value other than -1 "has performance implications when I/O threads are used due to additional tracking overhead".
# Give up large-reply logging in exchange for the 9.0.0 reply path
valkey-cli CONFIG SET commandlog-reply-larger-than -1
valkey-cli CONFIG REWRITE # only if you manage valkey.conf this wayInterpretation: this disables only the large-reply log. slow and large-request keep working. On 9.0.1 the setting does not remove the cost, so move to the newest 9.0 patch first.
COMMANDLOG large-reply: on versus off
Valkey 9.0.2 and later: switch settings to see what you trade (on 9.0.1, every setting behaves like "On")
On: 1048576 (default)
| Item | Behavior |
|---|---|
| Copy-avoided replies in the large-reply log | Logged |
| Byte counting on the copy-avoidance path | Every reply, on the main thread |
| Read throughput in the PR #3086 benchmark | About 1.66M requests per second |
| What you give up | Throughput, the 9.0.1 cost |
Off: -1
| Item | Behavior |
|---|---|
| Copy-avoided replies in the large-reply log | Not logged: the log is off |
| Byte counting on the copy-avoidance path | Skipped on the main thread; counted in the I/O thread |
| Read throughput in the PR #3086 benchmark | About 2.19M requests per second |
| What you give up | Large-reply entries |
Source of the numbers: one contributor benchmark in PR #3086, run on the development branch with io-threads 9 and a read workload. Treat the gap as a direction, and measure your own.
Keep large-reply on while you hunt for oversized values, then decide. With seven or more I/O threads, copy avoidance covers every raw string, so the default threshold makes every GET pay the counting cost. If the instance has headroom, keep the log as a guard against new large values. Either way, keep large-request on: it catches the writes that create big values, and the 9.0.2 note ties no cost to it.
How do you find the large values on a running instance?
Start with the logs, because they name the keys that clients actually read and write. Then scan the keyspace for values nobody has touched yet. Scan a replica where you can, because a scan is still extra load. The sibling Valkey latency spike runbook places this step inside a wider incident checklist.
# Largest key per type by memory, throttled: sleep 0.01 s every 100 SCAN calls
valkey-cli -h replica-02 --memkeys -i 0.01
# Largest key per type by element count (strings by STRLEN bytes)
valkey-cli -h replica-02 --bigkeys -i 0.01Interpretation: both modes print only the single biggest key per type, plus counts and averages. --memkeys asks MEMORY USAGE for each key, and --memkeys-samples sets how many nested elements it samples. In a cluster, run the scan against every primary or one replica per shard, because each node holds only its own slots.
To list every string above a size, not just the biggest, use a small SCAN loop.
# Strings over 1 MiB on one node. SCAN returns batches; STRLEN is O(1).
valkey-cli -h replica-02 --scan --count 1000 | while read -r key; do
len=$(valkey-cli -h replica-02 STRLEN "$key" 2>/dev/null)
case "$len" in ''|*[!0-9]*) continue ;; esac
[ "$len" -gt 1048576 ] && echo "$len $key"
done | sort -rn | head -50Interpretation: STRLEN returns 0 for a key that expired during the scan and a WRONGTYPE error for other types; the case line skips anything that is not a number. This opens one connection per key, which is slow but light; for millions of keys, pipeline the calls in your own client or run the scan in a script. Key names that contain newlines will break the loop.
valkey-cli MEMORY USAGE report:pdf:2026-11
valkey-cli MEMORY USAGE session:9f2c SAMPLES 0
valkey-cli OBJECT ENCODING report:pdf:2026-11Interpretation: MEMORY USAGE counts the value plus its overhead. For nested types, it samples 5 elements by default, and SAMPLES 0 reads them all, which is itself slow on a big collection. raw from OBJECT ENCODING means the string can take the copy-avoidance path.
How should you redesign values that are too large?
Copy avoidance makes large replies cheaper for the main thread, not smaller. Wire time, replication traffic, and the memory a slow client holds still grow with value size, so the lasting fix is in the data model.
- Chunk the value. Store a 10 MB object as, for example, 40 keys of 256 KB plus a small manifest key. Readers fetch chunks in a pipeline, so no single reply holds the write turn for long. In a cluster, a hash tag such as
{report:2026-11}:chunk:7keeps every chunk on one shard, soMGETworks on them. - Read only what you need. If clients read parts of a blob,
GETRANGEreturns a byte range without sending the whole value. If they read fields, store a hash and read the fields withHMGET. - Compress in the client. Valkey does not compress values in memory. JSON and text often compress well with zstd or LZ4 in the application, which cuts memory, wire time, and replication bytes together. Measure the client CPU cost first.
- Move blobs out. Put files, rendered pages, and model artifacts in object storage, and keep the object key, size, and ETag in Valkey.
- Cap sizes at the edge. Lowering
proto-max-bulk-lenmakes the server reject oversized strings. Its minimum is 1 MB, and it applies to every client, including the ones that legitimately write large values, so test it on staging first.
# Chunked write of one object; the hash tag keeps every chunk on one shard
split -b 262144 -d report.pdf chunk.
n=0; for f in chunk.*; do
valkey-cli -x SET "{report:2026-11}:chunk:$n" < "$f"; n=$((n+1))
done
valkey-cli HSET "{report:2026-11}:manifest" chunks "$n" bytes "$(stat -c %s report.pdf)"Interpretation: this shell version shows the layout only; an application would write the chunks in a MULTI block or a pipeline and write the manifest last, so readers never see a partial object. stat -c is GNU syntax; on macOS use stat -f %z.
Find and fix large values: a runbook
Seven steps; none needs downtime, and steps 3 and 4 add read load
-
Record the version and thread count
The copy-avoidance threshold and the cost of the large-reply log both depend on them.
valkey-cli INFO server | grep valkey_version valkey-cli CONFIG GET io-threads -
Make sure the size logs are on
Both default to 1 MiB. Raise the max length so a busy hour does not push out the entries you need.
valkey-cli CONFIG SET \ commandlog-reply-larger-than 1048576 valkey-cli CONFIG SET \ commandlog-large-reply-max-len 512 -
Read the logs after a busy period
Group entries by key prefix and client name to find the feature that owns the large values.
valkey-cli COMMANDLOG GET 100 large-reply valkey-cli COMMANDLOG GET 100 large-request -
Scan a replica for values nobody read yet
Throttle the scan, and repeat it per shard in a cluster.
valkey-cli -h replica-02 --memkeys -i 0.01 -
Measure the p99.9 baseline
Use client-side percentiles from your application, or a staging benchmark with the large values present.
-
Change the data model
Chunk, compress in the client, or move the blob to object storage. Deploy readers that understand the new layout before the writers change.
-
Re-measure, then decide on the log
If p99.9 is back and the instance is throughput-bound on 9.0.2 or later, consider turning the large-reply log off.
valkey-cli CONFIG SET \ commandlog-reply-larger-than -1
Main risk: changing writers before readers. Old readers that GET the original key will see nothing once a writer switches to chunks, so roll out readers that understand both layouts first.
Which Valkey version should you run for large values?
Run 9.0.2 or later for copy avoidance with the option to switch the large-reply cost off, and in practice the newest patch on your branch. As of November 2026, those are 9.1.3 (26 October 2026) and 9.0.7 (27 October 2026). 8.1 has COMMANDLOG but copies every reply.
One copy-avoidance fix is newer than both. The 9.2.0-rc2 release notes (4 November 2026) list "Fix a stack buffer overflow when replying with hundreds of empty strings (e.g. MGET) while reply copy avoidance is active (#5151)". The pull request says the write path sized an array for three iovec entries per string, while an empty string uses two, and that on a normal build the result is a failed writev and a dropped client connection. The 9.1.3 source still has the old sizing, and no 9.1 or 9.0 release note lists the fix yet.
valkey-cli INFO server | grep -E 'valkey_version|redis_version'
valkey-cli CONFIG GET io-threadsInterpretation: the exposure needs copy avoidance on empty strings, which in 9.1.3 means io-threads of 7 or more, since below that copy avoidance starts at 16 KB or 64 KB. If you run 7 or more I/O threads and send large MGETs over keys that can hold empty strings, watch the next 9.1 and 9.0 patch notes for #5151. 9.2 is a release candidate, so do not move production to it for this fix.
The 16 KB, 64 KB, and 7-thread rules come from source, not documentation, and they are hidden configs. Confirm them with CONFIG GET on your build after each upgrade, especially when moving to 9.2.
Frequently asked questions
Does Valkey reply copy avoidance need io-threads?
No. In the Valkey 9.0 and 9.1 source, copy avoidance applies to raw string replies of 16 KB or more when io-threads is 1, to 64 KB or more with 2 to 6 io-threads, and to any size at 7 or more. I/O threads add the most benefit, because then the write also leaves the main thread. The thresholds are hidden configs, so confirm them on your build.
Why does my Valkey p99.9 spike when clients read large keys?
Valkey runs commands on one main thread and writes each client a little over 64 KB per pass. A multi-MB reply takes many passes and holds the network for milliseconds, so small requests queue behind it. Before 9.0, the main thread also copied the whole value into the reply buffer. Look for the keys with COMMANDLOG GET large-reply and valkey-cli --memkeys.
Should I set commandlog-reply-larger-than to -1?
Only on Valkey 9.0.2 or later, and only if you are throughput-bound and do not need large-reply entries. Since 9.0.1, Valkey counts reply bytes on the copy-avoidance path whenever the log is on, which costs throughput with I/O threads. Setting -1 skips that counting on the main thread and disables only the large-reply log, not the slow or large-request logs.
How do I find big keys in Valkey without slowing production?
Read COMMANDLOG large-reply and large-request entries first, since they cost nothing extra to read. Then run valkey-cli --memkeys or --bigkeys with -i 0.01 against a replica, which throttles the SCAN. In a cluster, scan every shard. Use MEMORY USAGE or STRLEN on single keys. Avoid MEMORY USAGE with SAMPLES 0 on large collections in production.
Key takeaways
- Large values hurt the tail, not the median: watch p99.9 from the client, because p50 and even p99 can look normal.
- Valkey 9.0 reply copy avoidance removes the main-thread memcpy for qualifying string reads (GET, GETEX, GETDEL, GETSET, SET GET, and MGET), with thresholds set by io-threads.
- COMMANDLOG large-request and large-reply (8.1 and later) both default to 1 MiB; on 9.0.2 and later,
commandlog-reply-larger-than -1trades the large-reply log for throughput. - Find values with the logs first, then a throttled
--memkeysscan on a replica. - Fix the data model: chunk, compress in the client, or move blobs to object storage, and roll out readers before writers.
Tune Valkey for large values with JusDB
JusDB's Valkey performance tuning team can review your COMMANDLOG and big-key scans, measure tail latency under your real value sizes, and plan changes to io-threads, logging, and the data model, including a chunking or object-storage layout your application can roll out safely.
Bring your Valkey version, io-threads setting, client-side latency percentiles, and a sample of the large keys. Your team approves every production change.