A Valkey primary disappears, the replica takes over, and the log ends with Valkey 9.1.3 crashed by signal: 11 followed by several hundred lines between === VALKEY BUG REPORT START === and === VALKEY BUG REPORT END ===. What you do next decides whether the crash can be explained. Restart too fast and the core dump is lost. Copy the core to a laptop and a copy of production data has left its controls. The upstream Valkey debugging guide still shows a macOS backtrace and only covers attaching GDB before a crash. This post covers what Valkey 9.1 writes when it crashes, how to capture a core safely, and what to send upstream.
- The crash report is written to the Valkey log (the journal under systemd). It holds the stack trace, registers,
INFO all, the client list, the current command and a fast memory test. hide-user-data-from-log yesis the default since 8.0. It hides client names, ACL usernames, command arguments and key names, but not client IP addresses, and it never touches the core dump.- Capture cores with systemd-coredump, a
LimitCORE=infinitydrop-in, size limits that fit the instance, and storage off the Valkey data volume. Treat every core as a copy of the dataset. - Keep
enable-debug-command noin production. It is immutable at runtime, and mostDEBUGsubcommands crash, stall or change the server. - Upgrade to the newest patch first (9.1.3, 9.0.7, 8.1.11 or 8.0.12 as of November 2026). Then file the crash with the full report, a gdb backtrace and your build details, but never the core itself.
Scope and review date: November 2026. This post covers self-managed Valkey 9.1.3 on Linux with systemd, and the official valkey/valkey Docker images, with notes for 9.0, 8.1 and 8.0. Config names and defaults were checked against the 9.1.3 valkey.conf and source code, and the systemd settings against the systemd manual pages. The commands were not run in a lab, so rehearse them on staging.
What does a Valkey crash report contain?
When Valkey 9.1 receives SIGSEGV, SIGBUS, SIGFPE, SIGILL or SIGABRT, its signal handler writes a bug report to the log and then re-raises the signal with the default action, so the kernel can still write a core. The handler is installed only while crash-log-enabled is yes, which is the default. Assertion failures and panics write the same report and then abort the process.
Find where the report went before you restart anything:
valkey-cli CONFIG GET logfile # on a replica or the restarted node
journalctl -u valkey-server --since "-2h" --no-pager \
| sed -n '/BUG REPORT START/,/BUG REPORT END/p' > crash-report.txtInterpretation: an empty logfile means standard output, which systemd sends to the journal. The 9.1.3 valkey.conf warns that standard output plus daemonize yes sends logs to /dev/null, which loses the report.
A Valkey 9.1.3 crash report, trimmed
Illustrative output shaped like a 9.1.3 report, with log prefixes and most lines removed; the addresses and client are invented
Read it top down: signal and address, the thread marked *, then the current command. Section names match the 9.1.3 debug.c.
The sections appear in this order in 9.1.3. An assertion starts with === ASSERTION FAILED === and the failing file:line, and a panic starts with Guru Meditation; both skip the register dump.
| Section | What it tells you |
|---|---|
| Header | Version, signal number, si_code, faulting address; Killed by PID if another process sent the signal |
STACK TRACE | One block per thread (main, io_thd_N, bio_*); * marks the crashing thread |
REGISTERS | CPU registers at the fault (signal crashes only) |
INFO OUTPUT | INFO all, plus CLUSTER NODES in cluster mode |
CLIENT LIST OUTPUT | Every connected client, with the last command each one ran |
CURRENT / EXECUTING CLIENT INFO | The client and command being processed when it crashed |
MODULES INFO, CONFIG DEBUG OUTPUT | Loaded modules and a few configs such as io-threads |
FAST MEMORY TEST | A non-destructive RAM check on Linux, skipped if crash-memcheck-enabled no |
Default builds symbolize the stack with glibc backtrace(), so you get function names and offsets but no file and line numbers. Alpine (musl) has no execinfo. In the 9.1.3 source the crash handler collects stack traces only where it exists, and the libbacktrace fallback for musl (#3581) is listed only in the 9.2.0-rc1 release notes. Expect a 9.1.x Alpine crash report without a stack trace, and rely on the core.
Was it a Valkey bug, an OOM kill or bad memory?
Not every dead Valkey process is a Valkey bug. Classify it first.
From a Valkey crash to a useful bug report
Follow the numbers; the dashed path is the shortcut when the stack trace alone matches a fixed bug
Core stays inside: steps 3 and 4 happen on a controlled host. Only the reviewed report and backtrace leave it.
Three checks separate the causes:
# 1. Did the kernel OOM killer send SIGKILL? (no crash report is possible)
journalctl -k --since "-2h" | grep -iE "out of memory|killed process"
# 2. Did another process send the signal?
grep -E "Killed by PID|Crashed running signal handler" crash-report.txt
# 3. Did the fast memory test find anything?
grep -E "MEMORY ERROR DETECTED|Fast memory test PASSED" crash-report.txtInterpretation: SIGKILL cannot be caught, so an OOM kill leaves only a kernel log line. Killed by PID means a person or tool sent the signal. Crashed running signal handler means a second crash cut the report short. MEMORY ERROR DETECTED points at hardware.
A passed fast test does not clear the RAM. If one host crashes in unrelated functions, run the longer test from the problems page with Valkey stopped:
valkey-server --test-memory 4096 # size in megabytes
valkey-server --check-system # the END banner also suggests this checkInterpretation: both exit after the test without starting a server.
What does hide-user-data-from-log remove from the crash report?
hide-user-data-from-log was added in 8.0.0-rc2 (#877) and defaults to yes. The 9.1.3 valkey.conf says it covers keys, values, client names and ACL usernames logged by assertions and other error cases. In the crash report, that works out as follows.
Crash report fields with redaction on and off
Switch the setting to see which parts of a 9.1.3 crash report carry user data
hide-user-data-from-log yes (default)
| Part of the report | What is written |
|---|---|
| Stack trace and registers | Function names, addresses, CPU registers |
| Raw stack memory | Skipped, with a notice in the log |
| INFO output | INFO all: versions, memory, replica IPs, keyspace counts |
| Client list: name and user | *redacted* |
| Client list: addresses | Client IP addresses and ports |
| Current command arguments | Command name, then each argument as a byte count |
| Key found in the database | key '*redacted*', plus type and encoding |
hide-user-data-from-log no
| Part of the report | What is written |
|---|---|
| Stack trace and registers | Function names, addresses, CPU registers |
| Raw stack memory | 16 words of stack, which can hold user data |
| INFO output | INFO all: versions, memory, replica IPs, keyspace counts |
| Client list: name and user | Client names and ACL usernames in clear |
| Client list: addresses | Client IP addresses and ports |
| Current command arguments | Full keys and values (it stops after an AUTH command name) |
| Key found in the database | The key name in clear, plus type and encoding |
Never redacted: client IP addresses, INFO output, cluster node addresses, and the core dump. Review the report before you paste it anywhere public.
The redaction has had gaps. 9.1.1 redacts key names from more log messages (#3872), and 9.1.3, 9.0.7, 8.1.11 and 8.0.12 redact error text that could echo arguments into the log (#4815). On older patches, assume the log can hold keys and values. The other paths data takes out of Valkey, such as MONITOR and COMMANDLOG, are covered in where Valkey leaks secrets and personal data.
valkey-cli CONFIG GET hide-user-data-from-log
valkey-cli CONFIG GET crash-log-enabled
valkey-cli CONFIG GET crash-memcheck-enabledInterpretation: all three default to yes and can change at runtime. Leave them on. The 9.1.3 valkey.conf says crash-log-enabled no can give cleaner core dumps and crash-memcheck-enabled no a faster exit; use those only for a deliberate reproduction.
How do you enable core dumps for Valkey safely?
A core is an image of the process memory. It holds the dataset and can hold secrets kept in memory, such as masterauth and TLS key material. The debugging guide warns that "by sending the core file you are sending your data".
Enable and capture Valkey core dumps on a systemd host
Seven steps; step 7 crashes a staging server on purpose and must never run in production
-
Check where the kernel sends cores
A pattern starting with
|pipes to a handler such as systemd-coredump. A plain name such ascorewrites into the process working directory, which for Valkey is itsdir, the data directory.cat /proc/sys/kernel/core_pattern -
Size systemd-coredump for the instance
Defaults: 32G for
ExternalSizeMaxandProcessSizeMaxon 64-bit, andMaxUseof 10% of the disk capped at 4GiB. Set them above the instance's memory; the values below are examples.mkdir -p /etc/systemd/coredump.conf.d cat > /etc/systemd/coredump.conf.d/valkey.conf <<'EOF' [Coredump] ExternalSizeMax=80G ProcessSizeMax=80G MaxUse=200G EOF -
Put cores on their own disk and restrict access
Cores land in
/var/lib/systemd/coredump. Keep that off the Valkey data volume, limit it to the on-call group, and give it the retention of a backup.df -h /var/lib/systemd/coredump ls -ld /var/lib/systemd/coredump -
Raise the core limit and cap restarts
Add
LimitCORE=infinityand the restart limits shown in the restart-loop section below. The drop-in applies from the next start.systemctl edit valkey-server systemctl daemon-reload -
Raise the limit on the running process
Optional.
prlimitchanges the running server, so you do not need a restart to be ready for the next crash.prlimit --pid "$(pidof valkey-server)" \ --core=unlimited -
Verify the limit
The soft limit must not be 0. systemd also records it in the journal as
COREDUMP_RLIMIT=with each crash.grep "Max core file size" \ /proc/"$(pidof valkey-server)"/limits -
Test the pipeline on staging
Staging only: crashes the serverNeeds
enable-debug-command localin the staging config and a restart. Then confirm a core was stored.valkey-cli DEBUG SEGFAULT coredumpctl list valkey-server
Main risk: every stored core is a copy of the dataset outside your backup controls. Rollback is deleting the two drop-ins and running systemctl daemon-reload.
When a crash happens, extract the core and look at its metadata on the same host:
coredumpctl info valkey-server # newest crash: signal, PID, limits
coredumpctl -o /secure/valkey-41827.core dump 41827 # write one core to a fileInterpretation: info shows the signal and executable. A core larger than ExternalSizeMax is not stored, and one larger than ProcessSizeMax is stored without a stack trace (coredump.conf). If list shows the crash but no file, recheck step 2.
The process does not finish exiting until the memory test has run and the core is written. On a large instance that can add minutes before systemd restarts it. Without a replica, choose between the core and the faster restart.
How do core dumps work for Valkey in Docker and Kubernetes?
kernel.core_pattern belongs to the host kernel. The core(5) manual page says a piped handler "runs in the initial namespaces", so with systemd-coredump on the host, a container's core lands on the host. With a plain file pattern, it lands in the container's working directory, which in the official image is /data, next to dump.rdb.
# Docker: raise the core limit for one container
docker run -d --name valkey --ulimit core=-1 valkey/valkey:9.1.3
# On the host: read the limit the container's PID 1 started with
grep "Max core file size" \
/proc/"$(docker inspect -f '{{.State.Pid}}' valkey)"/limitsInterpretation: the image runs tini as PID 1, and valkey-server inherits its limit. A soft limit of 0 means no core. On Kubernetes, the limit comes from the container runtime on the node, so read it from /proc on the node in the same way.
On Kubernetes, a core in the container filesystem is gone once the kubelet replaces the container, and one on the data volume competes with RDB and AOF files for space. Collect cores at node level and pin the image digest.
How do you get debug symbols for a Valkey core?
gdb needs the exact binary that crashed and its symbols. A rebuild with different flags gives wrong frames.
- Source builds. The 9.1.3
src/Makefilecompiles with-O3, link-time optimization,-fno-omit-frame-pointerand-g -ggdb, andmake installdoes not strip. Your binary has symbols unless your packaging strips it. - Official Docker images. The 9.1 Debian Dockerfile runs the upstream
makewith no strip step. Neither image ships gdb, so analyse cores in a separate debug container. - Distribution packages. These are stripped. Debian publishes
valkey-tools-dbgsymin its debian-debug archive, Fedora-family systems usednf debuginfo-install, and gdb can also fetch symbols from the distributions' debuginfod servers. - Reproduction builds.
make noopt(-O0),make USE_LIBBACKTRACE=yes(9.1 and later, adds file and line to crash report traces) andmake SANITIZER=address. Use them on staging.
file "$(command -v valkey-server)" # want: with debug_info, not stripped
valkey-server --version # v=, sha=, malloc=, build= identify the build
docker image inspect --format '{{index .RepoDigests 0}}' valkey/valkey:9.1.3Interpretation: record all three with the core. The sha and build fields correspond to redis_git_sha1 and redis_build_id in the report's INFO output (still named that way in 9.1.3). If they differ, you have the wrong binary.
How do you read a Valkey core with gdb?
Open the core with the binary that produced it. Through systemd-coredump, coredumpctl debug does that for you.
coredumpctl debug valkey-server
# or, with an extracted core
gdb /usr/bin/valkey-server /secure/valkey-41827.core(gdb) bt # the crashing thread
(gdb) thread apply all bt # every thread: main, io_thd_N, bio_*
(gdb) info registers
(gdb) frame 2 # move into the caller you care about
(gdb) info locals # local variables: may print keys and valuesInterpretation: the top frames show where the fault happened; the cause is often a few frames up. In an optimized build many locals show as <optimized out>, which is normal.
For the issue, produce a plain backtrace file and read it before you share it:
gdb -batch -ex 'thread apply all bt' -ex 'info registers' \
/usr/bin/valkey-server /secure/valkey-41827.core > valkey-41827-bt.txtInterpretation: bt prints function arguments, and bt full adds locals. Both can contain key names or values. Keep bt full for private exchanges with a maintainer.
The upstream guide says attaching gdb costs no performance. Valkey is stopped while gdb attaches, at every breakpoint, and for a whole gcore run, so do it on a replica or a reproduction host.
Which DEBUG subcommands are safe to run in production?
Very few. DEBUG is a protected command. enable-debug-command defaults to no, accepts no, yes or local, and is immutable, so changing it takes a config file edit and a restart. While it is blocked, the server replies DEBUG command not allowed. followed by those instructions. The 9.1.3 DEBUG HELP subcommands fall into three kinds:
| Kind | Subcommands | In production |
|---|---|---|
| Crash or stall the server | SEGFAULT, PANIC, ASSERT, OOM, CRASH-AND-RECOVER, RESTART, SLEEP, PAUSE-CRON | Never |
| Change data, replication or internals | RELOAD, LOADAOF, POPULATE, REPLICATE, CHANGE-REPL-ID, SET-ACTIVE-EXPIRE, SET-SKIP-CHECKSUM-VALIDATION, LEAK | Never |
| Read internal state | OBJECT, SDSLEN, LISTPACK, QUICKLIST, HTSTATS, STRUCTSIZE, DIGEST-VALUE, DIGEST | Only if a maintainer asks; DIGEST hashes the whole dataset on the main thread and blocks it |
If a maintainer needs a read-only subcommand, set enable-debug-command local, restart during a planned failover, and allow it for one ACL user:
valkey-cli CONFIG GET enable-debug-command
valkey-cli ACL SETUSER debug-reader on '>use-a-long-random-password' '+debug|object' allkeysInterpretation: the config decides whether DEBUG runs at all, and the ACL which subcommands a user may call; the 9.1.3 ACL code allows DEBUG OBJECT alone through first-argument rules. Undo both afterwards.
How do you stop a crashing Valkey from restart-looping?
The upstream example unit, utils/systemd-valkey_server.service, has no Restart= line, so a crashed server stays down. Distribution units and Helm charts differ. The danger case is a crash caused by data loaded at startup: every restart replays the same RDB or AOF and crashes again. The 9.2.0-rc2 notes list one such bug: FUNCTION LOAD aborting the server when a library calls server.log() at its top level, including when loading from RDB or AOF (#5114).
systemctl show valkey-server -p Restart -p RestartUSec \
-p StartLimitBurst -p StartLimitIntervalUSecInterpretation: systemd's defaults are a 100 ms restart delay and a start limit of 5 starts in 10 seconds (systemd-system.conf). A Valkey that takes 30 seconds to load its data never reaches that limit, so it keeps looping and writes a core on every pass.
This drop-in allows at most 3 starts in 10 minutes, 10 seconds apart. At the limit, the unit stays failed:
# /etc/systemd/system/valkey-server.service.d/override.conf
[Unit]
StartLimitIntervalSec=600
StartLimitBurst=3
[Service]
LimitCORE=infinity
Restart=on-failure
RestartSec=10ssystemctl daemon-reload
systemctl reset-failed valkey-server # only after the cause is understoodInterpretation: the systemd.unit page says the limit applies to every start, including manual ones. If loaded data causes the crash, let the replica serve and rebuild the node from a replica sync or a known-good backup.
What should a Valkey crash bug report include?
First check whether it is already fixed. 9.1.2 and 9.1.3 alone fix crashes in module timers, ACL file loading, RDMA with I/O threads and blocked clients. Search for the top function names from the crashing thread:
gh search issues --repo valkey-io/valkey "zsetAdd"
gh release view 9.1.3 --repo valkey-io/valkey | grep -i crashInterpretation: a match in a newer patch on your branch means upgrade first. As of November 2026 those are 9.1.3, 9.0.7, 8.1.11 and 8.0.12. No match means a new report.
The Valkey crash report template asks for the complete crash log plus a few preceding lines, the OS version, and steps to reproduce; module crashes go to the module's repository. Also add:
- the report from START to END, with client IPs replaced if needed, and the setting of
hide-user-data-from-log; valkey-server --version, how it was installed (package version or image digest), and the CPU architecture;- the gdb output of
thread apply all btandinfo registers, produced with matching symbols; - the relevant non-default config: persistence,
io-threads, cluster mode, modules, TLS (no passwords); - whether it repeats, what was running, and whether a replica crashed too.
Do not attach the core. Offer it privately if a maintainer asks and your data owner agrees.
If a client that has not authenticated can trigger the crash, report it to security@lists.valkey.io and not in a public issue (SECURITY.md). Under Valkey's 17 September 2026 security policy, crashes that need an authenticated client are fixed as ordinary bugs.
Frequently asked questions
Where does Valkey write its crash report?
Valkey writes the crash report to its normal log, between lines that read VALKEY BUG REPORT START and VALKEY BUG REPORT END. With an empty logfile setting under systemd, that is the journal. With syslog enabled, most of it also goes to syslog, but the stack trace is written only to the log file or standard output. With an empty logfile and daemonize set to yes, output goes to /dev/null and the report is lost.
Does a Valkey crash report contain customer data?
With the default hide-user-data-from-log yes, Valkey 9.1 hides command arguments, key names, client names and ACL usernames, and skips the raw stack dump. Client IP addresses, INFO output and cluster node addresses are still written. A core dump is never redacted and contains the whole dataset, so treat it like a backup.
Is it safe to run DEBUG commands on a production Valkey server?
Mostly no. Many DEBUG subcommands crash, pause or change the server, and DEBUG DIGEST blocks the main thread while it hashes the dataset. Valkey blocks DEBUG by default with enable-debug-command no, which needs a restart to change. Use DEBUG SEGFAULT only on staging to test core capture.
Why did Valkey die without writing a crash report?
The most common reason is the Linux OOM killer, which sends SIGKILL, a signal no process can catch. Check the kernel log for "Killed process". Other reasons are crash-log-enabled set to no, a daemonized server logging to standard output, or a second crash inside the signal handler, which leaves only a reduced report.
Should I attach a Valkey core dump to a GitHub issue?
No. GitHub issues are public, and a core dump holds the full dataset plus secrets kept in memory. Post the crash report, a gdb backtrace produced with matching symbols, the version string and your build details. Offer the core privately only if a maintainer asks for it and your data owner agrees.
Key takeaways
- Save the crash report from START to END before you restart, and make sure the log goes somewhere other than
/dev/null. - Rule out the OOM killer and bad RAM before treating the crash as a Valkey bug.
- Capture cores with systemd-coredump, sized for the instance, off the data volume and access-controlled, and never attach one to a public issue.
- Keep
enable-debug-command noin production, and cap restarts so a crash on load cannot loop. - Upgrade to the newest patch for your branch, then file new crashes with the report, a gdb backtrace and exact build details.
Get help with a Valkey crash from JusDB
JusDB's Valkey support team can help you triage a crash, keep the evidence safe, read the crash report and backtrace, and match them against Valkey release notes. We can also plan the patch upgrade or mitigation and set up core capture and restart limits before the next incident.
Bring the crash report, your Valkey version and build source, and your topology. Your team approves every production change.