JusDB
Free DB Audit
NoSQL Databases

When Valkey Crashes: Reading the Crash Report, Capturing a Core Safely, Filing a Useful Bug

Read a Valkey 9.1 crash report, see what hide-user-data-from-log hides, capture core dumps safely with systemd or Docker, get symbols, use gdb, and file a useful bug.

JusDB Team
Published November 8, 2026
Updated November 10, 2026
20 min read

A Valkey primary disappears, the replica takes over, and the log ends with Valkey 9.1.3 crashed by signal: 11 followed by several hundred lines between === VALKEY BUG REPORT START === and === VALKEY BUG REPORT END ===. What you do next decides whether the crash can be explained. Restart too fast and the core dump is lost. Copy the core to a laptop and a copy of production data has left its controls. The upstream Valkey debugging guide still shows a macOS backtrace and only covers attaching GDB before a crash. This post covers what Valkey 9.1 writes when it crashes, how to capture a core safely, and what to send upstream.

TL;DR
  • The crash report is written to the Valkey log (the journal under systemd). It holds the stack trace, registers, INFO all, the client list, the current command and a fast memory test.
  • hide-user-data-from-log yes is the default since 8.0. It hides client names, ACL usernames, command arguments and key names, but not client IP addresses, and it never touches the core dump.
  • Capture cores with systemd-coredump, a LimitCORE=infinity drop-in, size limits that fit the instance, and storage off the Valkey data volume. Treat every core as a copy of the dataset.
  • Keep enable-debug-command no in production. It is immutable at runtime, and most DEBUG subcommands crash, stall or change the server.
  • Upgrade to the newest patch first (9.1.3, 9.0.7, 8.1.11 or 8.0.12 as of November 2026). Then file the crash with the full report, a gdb backtrace and your build details, but never the core itself.

Scope and review date: November 2026. This post covers self-managed Valkey 9.1.3 on Linux with systemd, and the official valkey/valkey Docker images, with notes for 9.0, 8.1 and 8.0. Config names and defaults were checked against the 9.1.3 valkey.conf and source code, and the systemd settings against the systemd manual pages. The commands were not run in a lab, so rehearse them on staging.


What does a Valkey crash report contain?

When Valkey 9.1 receives SIGSEGV, SIGBUS, SIGFPE, SIGILL or SIGABRT, its signal handler writes a bug report to the log and then re-raises the signal with the default action, so the kernel can still write a core. The handler is installed only while crash-log-enabled is yes, which is the default. Assertion failures and panics write the same report and then abort the process.

Find where the report went before you restart anything:

bash
valkey-cli CONFIG GET logfile      # on a replica or the restarted node
journalctl -u valkey-server --since "-2h" --no-pager \
  | sed -n '/BUG REPORT START/,/BUG REPORT END/p' > crash-report.txt

Interpretation: an empty logfile means standard output, which systemd sends to the journal. The 9.1.3 valkey.conf warns that standard output plus daemonize yes sends logs to /dev/null, which loses the report.

A Valkey 9.1.3 crash report, trimmed

Illustrative output shaped like a 9.1.3 report, with log prefixes and most lines removed; the addresses and client are invented

ops@cache-01:~
$ journalctl -u valkey-server --since "-2h" --no-pager \ | sed -n '/BUG REPORT START/,/BUG REPORT END/p'=== VALKEY BUG REPORT START: Cut & paste starting from here ===# Valkey 9.1.3 crashed by signal: 11, si_code: 1# Accessing address: 0x18------ STACK TRACE ------41827 valkey-server *valkey-server *:6379(zsetAdd+0x1a1)[0x55e3c1a4b2f1]valkey-server *:6379(zaddGenericCommand+0x2c4)[0x55e3c1a4d914]------ INFO OUTPUT ------valkey_version:9.1.3------ CLIENT LIST OUTPUT ------id=7 addr=10.0.4.21:51734 name=*redacted* cmd=zadd user=*redacted*------ CURRENT CLIENT INFO ------argv[1]: 9 bytes------ FAST MEMORY TEST ------Fast memory test PASSED, however your memory can still be broken.=== VALKEY BUG REPORT END. Make sure to include from START to END. ===

Read it top down: signal and address, the thread marked *, then the current command. Section names match the 9.1.3 debug.c.

The sections appear in this order in 9.1.3. An assertion starts with === ASSERTION FAILED === and the failing file:line, and a panic starts with Guru Meditation; both skip the register dump.

SectionWhat it tells you
HeaderVersion, signal number, si_code, faulting address; Killed by PID if another process sent the signal
STACK TRACEOne block per thread (main, io_thd_N, bio_*); * marks the crashing thread
REGISTERSCPU registers at the fault (signal crashes only)
INFO OUTPUTINFO all, plus CLUSTER NODES in cluster mode
CLIENT LIST OUTPUTEvery connected client, with the last command each one ran
CURRENT / EXECUTING CLIENT INFOThe client and command being processed when it crashed
MODULES INFO, CONFIG DEBUG OUTPUTLoaded modules and a few configs such as io-threads
FAST MEMORY TESTA non-destructive RAM check on Linux, skipped if crash-memcheck-enabled no

Default builds symbolize the stack with glibc backtrace(), so you get function names and offsets but no file and line numbers. Alpine (musl) has no execinfo. In the 9.1.3 source the crash handler collects stack traces only where it exists, and the libbacktrace fallback for musl (#3581) is listed only in the 9.2.0-rc1 release notes. Expect a 9.1.x Alpine crash report without a stack trace, and rely on the core.

Was it a Valkey bug, an OOM kill or bad memory?

Not every dead Valkey process is a Valkey bug. Classify it first.

From a Valkey crash to a useful bug report

Follow the numbers; the dashed path is the shortcut when the stack trace alone matches a fixed bug

Valkey crash triage flow Step 1: Valkey crashes with a signal or an assertion. Step 2: the crash report is written to the log or the journal. Step 3: the kernel writes a core dump, captured by systemd-coredump. Step 4: the core is opened in gdb with matching debug symbols to get a full backtrace. Step 5: the backtrace and the crash report are compared against the release notes and existing issues. A dashed shortcut goes straight from the crash report to the release notes when the stack trace alone identifies a known bug. Step 6: either upgrade to the patch that fixes it, or file a GitHub issue with the report and backtrace. Crash signal or assert Crash report log or journal Core dump systemd-coredump gdb analysis symbols + backtrace Known bug? release notes, issues Upgrade or file patch or GitHub issue 1 2 3 4 5 6 trace matches the core stays on the host; only the report and backtrace go upstream

Core stays inside: steps 3 and 4 happen on a controlled host. Only the reviewed report and backtrace leave it.

Three checks separate the causes:

bash
# 1. Did the kernel OOM killer send SIGKILL? (no crash report is possible)
journalctl -k --since "-2h" | grep -iE "out of memory|killed process"

# 2. Did another process send the signal?
grep -E "Killed by PID|Crashed running signal handler" crash-report.txt

# 3. Did the fast memory test find anything?
grep -E "MEMORY ERROR DETECTED|Fast memory test PASSED" crash-report.txt

Interpretation: SIGKILL cannot be caught, so an OOM kill leaves only a kernel log line. Killed by PID means a person or tool sent the signal. Crashed running signal handler means a second crash cut the report short. MEMORY ERROR DETECTED points at hardware.

A passed fast test does not clear the RAM. If one host crashes in unrelated functions, run the longer test from the problems page with Valkey stopped:

bash
valkey-server --test-memory 4096   # size in megabytes
valkey-server --check-system       # the END banner also suggests this check

Interpretation: both exit after the test without starting a server.

What does hide-user-data-from-log remove from the crash report?

hide-user-data-from-log was added in 8.0.0-rc2 (#877) and defaults to yes. The 9.1.3 valkey.conf says it covers keys, values, client names and ACL usernames logged by assertions and other error cases. In the crash report, that works out as follows.

Crash report fields with redaction on and off

Switch the setting to see which parts of a 9.1.3 crash report carry user data

hide-user-data-from-log yes (default)

Part of the reportWhat is written
Stack trace and registersFunction names, addresses, CPU registers
Raw stack memorySkipped, with a notice in the log
INFO outputINFO all: versions, memory, replica IPs, keyspace counts
Client list: name and user*redacted*
Client list: addressesClient IP addresses and ports
Current command argumentsCommand name, then each argument as a byte count
Key found in the databasekey '*redacted*', plus type and encoding

hide-user-data-from-log no

Part of the reportWhat is written
Stack trace and registersFunction names, addresses, CPU registers
Raw stack memory16 words of stack, which can hold user data
INFO outputINFO all: versions, memory, replica IPs, keyspace counts
Client list: name and userClient names and ACL usernames in clear
Client list: addressesClient IP addresses and ports
Current command argumentsFull keys and values (it stops after an AUTH command name)
Key found in the databaseThe key name in clear, plus type and encoding

Never redacted: client IP addresses, INFO output, cluster node addresses, and the core dump. Review the report before you paste it anywhere public.

The redaction has had gaps. 9.1.1 redacts key names from more log messages (#3872), and 9.1.3, 9.0.7, 8.1.11 and 8.0.12 redact error text that could echo arguments into the log (#4815). On older patches, assume the log can hold keys and values. The other paths data takes out of Valkey, such as MONITOR and COMMANDLOG, are covered in where Valkey leaks secrets and personal data.

bash
valkey-cli CONFIG GET hide-user-data-from-log
valkey-cli CONFIG GET crash-log-enabled
valkey-cli CONFIG GET crash-memcheck-enabled

Interpretation: all three default to yes and can change at runtime. Leave them on. The 9.1.3 valkey.conf says crash-log-enabled no can give cleaner core dumps and crash-memcheck-enabled no a faster exit; use those only for a deliberate reproduction.

How do you enable core dumps for Valkey safely?

A core is an image of the process memory. It holds the dataset and can hold secrets kept in memory, such as masterauth and TLS key material. The debugging guide warns that "by sending the core file you are sending your data".

Enable and capture Valkey core dumps on a systemd host

Seven steps; step 7 crashes a staging server on purpose and must never run in production

  1. Check where the kernel sends cores

    A pattern starting with | pipes to a handler such as systemd-coredump. A plain name such as core writes into the process working directory, which for Valkey is its dir, the data directory.

    cat /proc/sys/kernel/core_pattern
  2. Size systemd-coredump for the instance

    Defaults: 32G for ExternalSizeMax and ProcessSizeMax on 64-bit, and MaxUse of 10% of the disk capped at 4GiB. Set them above the instance's memory; the values below are examples.

    mkdir -p /etc/systemd/coredump.conf.d cat > /etc/systemd/coredump.conf.d/valkey.conf <<'EOF' [Coredump] ExternalSizeMax=80G ProcessSizeMax=80G MaxUse=200G EOF
  3. Put cores on their own disk and restrict access

    Cores land in /var/lib/systemd/coredump. Keep that off the Valkey data volume, limit it to the on-call group, and give it the retention of a backup.

    df -h /var/lib/systemd/coredump ls -ld /var/lib/systemd/coredump
  4. Raise the core limit and cap restarts

    Add LimitCORE=infinity and the restart limits shown in the restart-loop section below. The drop-in applies from the next start.

    systemctl edit valkey-server systemctl daemon-reload
  5. Raise the limit on the running process

    Optional. prlimit changes the running server, so you do not need a restart to be ready for the next crash.

    prlimit --pid "$(pidof valkey-server)" \ --core=unlimited
  6. Verify the limit

    The soft limit must not be 0. systemd also records it in the journal as COREDUMP_RLIMIT= with each crash.

    grep "Max core file size" \ /proc/"$(pidof valkey-server)"/limits
  7. Test the pipeline on staging

    Staging only: crashes the server

    Needs enable-debug-command local in the staging config and a restart. Then confirm a core was stored.

    valkey-cli DEBUG SEGFAULT coredumpctl list valkey-server

Main risk: every stored core is a copy of the dataset outside your backup controls. Rollback is deleting the two drop-ins and running systemctl daemon-reload.

When a crash happens, extract the core and look at its metadata on the same host:

bash
coredumpctl info valkey-server                       # newest crash: signal, PID, limits
coredumpctl -o /secure/valkey-41827.core dump 41827  # write one core to a file

Interpretation: info shows the signal and executable. A core larger than ExternalSizeMax is not stored, and one larger than ProcessSizeMax is stored without a stack trace (coredump.conf). If list shows the crash but no file, recheck step 2.

Large cores delay recovery

The process does not finish exiting until the memory test has run and the core is written. On a large instance that can add minutes before systemd restarts it. Without a replica, choose between the core and the faster restart.

How do core dumps work for Valkey in Docker and Kubernetes?

kernel.core_pattern belongs to the host kernel. The core(5) manual page says a piped handler "runs in the initial namespaces", so with systemd-coredump on the host, a container's core lands on the host. With a plain file pattern, it lands in the container's working directory, which in the official image is /data, next to dump.rdb.

bash
# Docker: raise the core limit for one container
docker run -d --name valkey --ulimit core=-1 valkey/valkey:9.1.3

# On the host: read the limit the container's PID 1 started with
grep "Max core file size" \
  /proc/"$(docker inspect -f '{{.State.Pid}}' valkey)"/limits

Interpretation: the image runs tini as PID 1, and valkey-server inherits its limit. A soft limit of 0 means no core. On Kubernetes, the limit comes from the container runtime on the node, so read it from /proc on the node in the same way.

On Kubernetes, a core in the container filesystem is gone once the kubelet replaces the container, and one on the data volume competes with RDB and AOF files for space. Collect cores at node level and pin the image digest.

How do you get debug symbols for a Valkey core?

gdb needs the exact binary that crashed and its symbols. A rebuild with different flags gives wrong frames.

  • Source builds. The 9.1.3 src/Makefile compiles with -O3, link-time optimization, -fno-omit-frame-pointer and -g -ggdb, and make install does not strip. Your binary has symbols unless your packaging strips it.
  • Official Docker images. The 9.1 Debian Dockerfile runs the upstream make with no strip step. Neither image ships gdb, so analyse cores in a separate debug container.
  • Distribution packages. These are stripped. Debian publishes valkey-tools-dbgsym in its debian-debug archive, Fedora-family systems use dnf debuginfo-install, and gdb can also fetch symbols from the distributions' debuginfod servers.
  • Reproduction builds. make noopt (-O0), make USE_LIBBACKTRACE=yes (9.1 and later, adds file and line to crash report traces) and make SANITIZER=address. Use them on staging.
bash
file "$(command -v valkey-server)"     # want: with debug_info, not stripped
valkey-server --version               # v=, sha=, malloc=, build= identify the build
docker image inspect --format '{{index .RepoDigests 0}}' valkey/valkey:9.1.3

Interpretation: record all three with the core. The sha and build fields correspond to redis_git_sha1 and redis_build_id in the report's INFO output (still named that way in 9.1.3). If they differ, you have the wrong binary.

How do you read a Valkey core with gdb?

Open the core with the binary that produced it. Through systemd-coredump, coredumpctl debug does that for you.

bash
coredumpctl debug valkey-server
# or, with an extracted core
gdb /usr/bin/valkey-server /secure/valkey-41827.core
bash
(gdb) bt                      # the crashing thread
(gdb) thread apply all bt     # every thread: main, io_thd_N, bio_*
(gdb) info registers
(gdb) frame 2                 # move into the caller you care about
(gdb) info locals             # local variables: may print keys and values

Interpretation: the top frames show where the fault happened; the cause is often a few frames up. In an optimized build many locals show as <optimized out>, which is normal.

For the issue, produce a plain backtrace file and read it before you share it:

bash
gdb -batch -ex 'thread apply all bt' -ex 'info registers' \
    /usr/bin/valkey-server /secure/valkey-41827.core > valkey-41827-bt.txt

Interpretation: bt prints function arguments, and bt full adds locals. Both can contain key names or values. Keep bt full for private exchanges with a maintainer.

Live attach stalls the server

The upstream guide says attaching gdb costs no performance. Valkey is stopped while gdb attaches, at every breakpoint, and for a whole gcore run, so do it on a replica or a reproduction host.

Which DEBUG subcommands are safe to run in production?

Very few. DEBUG is a protected command. enable-debug-command defaults to no, accepts no, yes or local, and is immutable, so changing it takes a config file edit and a restart. While it is blocked, the server replies DEBUG command not allowed. followed by those instructions. The 9.1.3 DEBUG HELP subcommands fall into three kinds:

KindSubcommandsIn production
Crash or stall the serverSEGFAULT, PANIC, ASSERT, OOM, CRASH-AND-RECOVER, RESTART, SLEEP, PAUSE-CRONNever
Change data, replication or internalsRELOAD, LOADAOF, POPULATE, REPLICATE, CHANGE-REPL-ID, SET-ACTIVE-EXPIRE, SET-SKIP-CHECKSUM-VALIDATION, LEAKNever
Read internal stateOBJECT, SDSLEN, LISTPACK, QUICKLIST, HTSTATS, STRUCTSIZE, DIGEST-VALUE, DIGESTOnly if a maintainer asks; DIGEST hashes the whole dataset on the main thread and blocks it

If a maintainer needs a read-only subcommand, set enable-debug-command local, restart during a planned failover, and allow it for one ACL user:

bash
valkey-cli CONFIG GET enable-debug-command
valkey-cli ACL SETUSER debug-reader on '>use-a-long-random-password' '+debug|object' allkeys

Interpretation: the config decides whether DEBUG runs at all, and the ACL which subcommands a user may call; the 9.1.3 ACL code allows DEBUG OBJECT alone through first-argument rules. Undo both afterwards.

How do you stop a crashing Valkey from restart-looping?

The upstream example unit, utils/systemd-valkey_server.service, has no Restart= line, so a crashed server stays down. Distribution units and Helm charts differ. The danger case is a crash caused by data loaded at startup: every restart replays the same RDB or AOF and crashes again. The 9.2.0-rc2 notes list one such bug: FUNCTION LOAD aborting the server when a library calls server.log() at its top level, including when loading from RDB or AOF (#5114).

bash
systemctl show valkey-server -p Restart -p RestartUSec \
    -p StartLimitBurst -p StartLimitIntervalUSec

Interpretation: systemd's defaults are a 100 ms restart delay and a start limit of 5 starts in 10 seconds (systemd-system.conf). A Valkey that takes 30 seconds to load its data never reaches that limit, so it keeps looping and writes a core on every pass.

This drop-in allows at most 3 starts in 10 minutes, 10 seconds apart. At the limit, the unit stays failed:

ini
# /etc/systemd/system/valkey-server.service.d/override.conf
[Unit]
StartLimitIntervalSec=600
StartLimitBurst=3

[Service]
LimitCORE=infinity
Restart=on-failure
RestartSec=10s
bash
systemctl daemon-reload
systemctl reset-failed valkey-server   # only after the cause is understood

Interpretation: the systemd.unit page says the limit applies to every start, including manual ones. If loaded data causes the crash, let the replica serve and rebuild the node from a replica sync or a known-good backup.

What should a Valkey crash bug report include?

First check whether it is already fixed. 9.1.2 and 9.1.3 alone fix crashes in module timers, ACL file loading, RDMA with I/O threads and blocked clients. Search for the top function names from the crashing thread:

bash
gh search issues --repo valkey-io/valkey "zsetAdd"
gh release view 9.1.3 --repo valkey-io/valkey | grep -i crash

Interpretation: a match in a newer patch on your branch means upgrade first. As of November 2026 those are 9.1.3, 9.0.7, 8.1.11 and 8.0.12. No match means a new report.

The Valkey crash report template asks for the complete crash log plus a few preceding lines, the OS version, and steps to reproduce; module crashes go to the module's repository. Also add:

  • the report from START to END, with client IPs replaced if needed, and the setting of hide-user-data-from-log;
  • valkey-server --version, how it was installed (package version or image digest), and the CPU architecture;
  • the gdb output of thread apply all bt and info registers, produced with matching symbols;
  • the relevant non-default config: persistence, io-threads, cluster mode, modules, TLS (no passwords);
  • whether it repeats, what was running, and whether a replica crashed too.

Do not attach the core. Offer it privately if a maintainer asks and your data owner agrees.

Some crashes are security issues

If a client that has not authenticated can trigger the crash, report it to security@lists.valkey.io and not in a public issue (SECURITY.md). Under Valkey's 17 September 2026 security policy, crashes that need an authenticated client are fixed as ordinary bugs.

Frequently asked questions

Where does Valkey write its crash report?

Valkey writes the crash report to its normal log, between lines that read VALKEY BUG REPORT START and VALKEY BUG REPORT END. With an empty logfile setting under systemd, that is the journal. With syslog enabled, most of it also goes to syslog, but the stack trace is written only to the log file or standard output. With an empty logfile and daemonize set to yes, output goes to /dev/null and the report is lost.

Does a Valkey crash report contain customer data?

With the default hide-user-data-from-log yes, Valkey 9.1 hides command arguments, key names, client names and ACL usernames, and skips the raw stack dump. Client IP addresses, INFO output and cluster node addresses are still written. A core dump is never redacted and contains the whole dataset, so treat it like a backup.

Is it safe to run DEBUG commands on a production Valkey server?

Mostly no. Many DEBUG subcommands crash, pause or change the server, and DEBUG DIGEST blocks the main thread while it hashes the dataset. Valkey blocks DEBUG by default with enable-debug-command no, which needs a restart to change. Use DEBUG SEGFAULT only on staging to test core capture.

Why did Valkey die without writing a crash report?

The most common reason is the Linux OOM killer, which sends SIGKILL, a signal no process can catch. Check the kernel log for "Killed process". Other reasons are crash-log-enabled set to no, a daemonized server logging to standard output, or a second crash inside the signal handler, which leaves only a reduced report.

Should I attach a Valkey core dump to a GitHub issue?

No. GitHub issues are public, and a core dump holds the full dataset plus secrets kept in memory. Post the crash report, a gdb backtrace produced with matching symbols, the version string and your build details. Offer the core privately only if a maintainer asks for it and your data owner agrees.

Key takeaways

  • Save the crash report from START to END before you restart, and make sure the log goes somewhere other than /dev/null.
  • Rule out the OOM killer and bad RAM before treating the crash as a Valkey bug.
  • Capture cores with systemd-coredump, sized for the instance, off the data volume and access-controlled, and never attach one to a public issue.
  • Keep enable-debug-command no in production, and cap restarts so a crash on load cannot loop.
  • Upgrade to the newest patch for your branch, then file new crashes with the report, a gdb backtrace and exact build details.

Get help with a Valkey crash from JusDB

JusDB's Valkey support team can help you triage a crash, keep the evidence safe, read the crash report and backtrace, and match them against Valkey release notes. We can also plan the patch upgrade or mitigation and set up core capture and restart limits before the next incident.

Bring the crash report, your Valkey version and build source, and your topology. Your team approves every production change.

Share this article

Database engineering notes

Articles like this one, in your inbox. No spam, unsubscribe anytime.

JusDB Team

Official JusDB content team

Keep reading

Moving a Multi-Database Valkey Instance into a 9.x Cluster

Move a Valkey instance that uses SELECT 1 to 15 into a 9.x cluster: keep or remap databases, copy each one, verify counts per database, and keep a way back.

Redis18 minNov 10, 2026
Read

Valkey Keyspace Notifications in Production: Expired Events, hexpired and Cluster Mode

Run Valkey keyspace notifications in production: why expired and hexpired events lag the TTL, how to subscribe on every cluster node, and when a stream is safer.

Redis20 minNov 10, 2026
Read

Valkey GLIDE in Production: AZ-Affinity Reads and Migrating from Legacy Clients

Configure Valkey GLIDE AZ-affinity reads on servers and clients, learn what GLIDE 2.5.3 does not support, and migrate from redis-py, Jedis, ioredis, or go-redis one service at a time.

Redis20 minNov 10, 2026
Read