Running Valkey on VMs: Clocksource, Huge Pages and a Latency Baseline

Valkey 9.1 changed how the server reads time, and on cloud VMs that change has already caused two production bugs. From 9.1.0, an x86-64 Linux build reads the CPU's timestamp counter (TSC) directly instead of calling clock_gettime. On hosts with an unsynchronized TSC, 9.1.0 and 9.1.1 could start with a frozen clock that stopped key expiration. On VMs whose kernel clocksource is kvm-clock or xen, releases before 9.1.3 could busy-spin the event loop and burn a full core. Neither raises an error. This post covers that clock change and the rest of the host baseline that sets Valkey's latency floor on a VM: transparent huge pages, swap, NUMA, interrupts, noisy neighbours, and valkey-cli --intrinsic-latency.

Scope and review date: November 2026. This post covers self-managed Valkey 9.1.x on x86-64 and arm64 Linux VMs, with notes on 9.0, 8.1, and the 9.2 release candidates. Versions, defaults, and messages were checked against the Valkey 9.1.3 source, the GitHub release notes, and vendor documentation on 10 November 2026. The commands were not run in a lab, so rehearse them on a staging host first.

What changed in how Valkey 9.1 reads the clock?

Valkey's event loop, timers, and serverCron all run on a monotonic clock. Up to 9.0, the default build used clock_gettime(CLOCK_MONOTONIC). The faster processor-clock path existed, but only if you compiled with USE_PROCESSOR_CLOCK; the 9.0.7 and 8.1.11 sources still carry that opt-in. Valkey 9.1.0-rc1 made it the default ("Enable hardware clock by default", #3103). The 9.1 announcement credits it with improving GET and SET performance by up to 3% overall, in Valkey's own tests.

In the 9.1.3 monotonic.c, the choice works like this:

• x86-64 Linux: Valkey reads current_clocksource. If it is tsc, Valkey calibrates the TSC against CLOCK_MONOTONIC over three 10 ms samples, checks for the constant_tsc CPU flag, and uses RDTSC. Otherwise it falls back to clock_gettime.
• arm64: Valkey reads the CNTVCT virtual counter directly. The kernel clocksource check does not apply on this path.
• Opting out: build with CFLAGS="-DNO_PROCESSOR_CLOCK" to always use clock_gettime.

# Which clock did this server pick? Run on every node.
valkey-cli INFO server | grep -E '^(valkey_version|monotonic_clock):'

Interpretation: X86 TSC @ <n> ticks/us means raw TSC, ARM CNTVCT @ <n> ticks/us means the arm64 counter, and POSIX clock_gettime means the fallback. The server also writes monotonic clock: ... to its log at startup. The INFO reference shows the field only in an example and does not describe it, as of November 2026.

Which Valkey 9.1 releases had clock bugs on VMs?

Two bugs, one in each direction, both on x86-64 Linux.

9.1.0 and 9.1.1: a frozen clock that stopped expiration

Issue #4345 describes hosts whose TSC is not synchronized across cores. If the calibrating thread moved to a core with a lower TSC during a sample, the unsigned subtraction wrapped, the multiplier became zero, and the clock read 0 for the life of the process. serverCron never ran, TIME stayed fixed, and key expiration stopped. The 9.1.2 release notes (31 August 2026) list the fix as #4346, which discards backwards samples and rejects a zero multiplier. These releases did not check the kernel clocksource at all; the #4707 pull request notes that the #4345 host ran on hpet after the kernel had marked its TSC unstable.

The reporter's wedged server showed monotonic_clock:X86 TSC @ inf ticks/us and returned the same TIME on every call, so two TIME calls a few seconds apart are a quick check.

9.1.0 to 9.1.2: a busy-spinning event loop on kvm-clock and xen

Issue #4691 reported a Sentinel on 9.1.2, on one AWS c6a.2xlarge host, running about 55,000 event loop cycles per second while a healthy node ran 67. The pull request explains why: Valkey computed the epoll_wait timeout from raw RDTSC, while the kernel timed the sleep with kvm-clock, which applies a paravirtual correction. After a vCPU migration the guest TSC could jump forward, the timeout came out as zero, and the loop spun.

Valkey 9.1.3 (26 October 2026) ships the fix, #4707: when current_clocksource is not tsc, Valkey uses clock_gettime(CLOCK_MONOTONIC), the same clock the kernel uses for the sleep. 9.1.3 is also marked upgrade urgency SECURITY for an unrelated cluster Pub/Sub fix, so there are two reasons to take it.

# On an idle node, cycles per second should be low and steady
valkey-cli INFO stats | grep -E 'instantaneous_eventloop_(cycles_per_sec|duration_usec)'
top -b -n 1 -p "$(pidof valkey-server)" | tail -2

Interpretation: the #4707 test plan expects an idle server to stay in the tens to low hundreds of cycles per second. Tens of thousands on an idle node, with valkey-server near 100% CPU, matches the spin. The fix is the upgrade.

How do you check the clocksource on a Linux VM?

The kernel exposes the current and available clocksources in sysfs. Read both, then compare them with what Valkey reports.

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksource
grep -o -w -E 'constant_tsc|nonstop_tsc|tsc_known_freq' /proc/cpuinfo | sort -u
dmesg | grep -i clocksource

Interpretation: the first line is what the kernel uses now; the second is what it could switch to. Without constant_tsc, Valkey will not use the TSC even on tsc. A kernel log line marking the TSC unstable means the kernel stopped trusting it on this host.

valkey-server --check-system; echo "exit=$?"

Interpretation: in 9.1.3 it runs slow-clocksource, xen-clocksource, overcommit, and THP, and prints OK, skipped, or WARNING with advice for each. The exit code is 1 if any check warns. Since 9.1.2 (#4272), slow-clocksource is skipped when Valkey uses a hardware clock, and its advice names only clocksources the host actually offers.

Should you switch a VM to the tsc clocksource?

Not by default. AWS's clock source article calls tsc best practice for all EC2 Intel and AMD instances, on both the Xen and Nitro hypervisors, and notes that Graviton processors use arch_sys_counter. On 9.1.3, switching to tsc returns Valkey to the raw TSC path and its speed-up, along with the risk both bugs came from: a TSC that is not stable on that host.

The #4707 pull request notes that kernels rate kvm-clock above tsc, so it is the usual default on KVM guests. Its author estimates the vDSO clock_gettime path at roughly 20 to 40 ns per call, against 7 to 15 ns for RDTSC. By that estimate, the difference is well under 1% of a core at 100,000 operations per second, and a few percent at about 1 million. These are estimates, not measurements.

If you do switch, change it at runtime, make it permanent on the kernel command line, and restart Valkey, which reads the clocksource only at startup.

# Runtime (reverts on reboot); then check dmesg for "Switched to clocksource tsc"
echo tsc | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource

# Permanent: add to GRUB_CMDLINE_LINUX in /etc/default/grub, then regenerate grub.cfg
#   clocksource=tsc
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Interpretation: AWS's example adds tsc=reliable as well. The kernel parameter documentation says tsc=reliable disables clocksource verification at runtime and the stability checks at boot. That removes the safety net that would catch an unstable TSC, so leave it out unless you have tested that host family.

Should transparent huge pages be never or madvise for Valkey?

Either is fine; always is not. The sources disagree on which of the two to recommend:

• The Valkey latency guide says THP "must be disabled" and gives echo never.
• The 9.1.3 startup warning and --check-system recommend echo madvise and say Valkey must be restarted after THP is set to "madvise" or "never".
• In 9.1.3, LATENCY DOCTOR still prints the echo never advice. 9.2.0-rc1 changes it to madvise (#3947) to match the startup check.

The kernel THP documentation explains why both work. With madvise, huge pages are used only inside regions that ask for them with MADV_HUGEPAGE, so a process that does not ask gets normal pages. The harm comes from always: after a fork for BGSAVE or a full sync, every write to a shared huge page copies the whole page, which adds latency and memory use.

cat /sys/kernel/mm/transparent_hugepage/enabled     # brackets mark the active value
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
grep AnonHugePages /proc/"$(pidof valkey-server)"/smaps_rollup
valkey-cli CONFIG GET disable-thp

Interpretation: always madvise [never] or always [madvise] never is what you want. A non-zero AnonHugePages for valkey-server means it already holds huge pages, and only a restart clears them. Persist the setting with transparent_hugepage=madvise on the kernel command line or a boot-time unit.

disable-thp defaults to yes and cannot be changed at runtime. When the kernel is on always, Valkey calls prctl(PR_SET_THP_DISABLE) for its own process and, if that succeeds, skips the warning; the kernel documents that the flag is inherited across fork. That is a safety net, not a reason to leave the host on always, because other processes on the host still get huge pages. For fork sizing, overcommit, and the copy-on-write side, see the sibling post on BGSAVE failures and fork latency.

How do swap and NUMA placement affect Valkey latency on a VM?

Swap. Valkey answers from memory, so a swapped-out page turns a microsecond read into a disk read. The latency guide's test: if every Swap: line in the process's smaps reads 0 kB, or only sporadic 4 kB entries, all is normal.

grep -E '^(VmSwap|Cpus_allowed_list|Mems_allowed_list)' /proc/"$(pidof valkey-server)"/status
vmstat 1 5     # watch si, so (swap in/out per second) and st (steal)

Interpretation: VmSwap is the anonymous memory of the process that is swapped out now. Non-zero si and so during normal traffic mean the host is short of memory. Size the instance for the dataset, buffers, and fork copy-on-write instead.

NUMA. Larger instances can expose more than one NUMA node. Memory on the remote node costs more to reach, and a process that drifts across nodes shows uneven latency. Valkey's performance dashboard methodology isolates the server and benchmark client on separate NUMA nodes to remove L3 cache contention.

lscpu | grep -i numa
numactl --hardware
# Example: keep a Valkey process and its memory on node 0
numactl --cpunodebind=0 --membind=0 valkey-server /etc/valkey/valkey.conf

Interpretation: with one NUMA node there is nothing to do. With two or more, binding each instance to one node keeps memory local. --membind is strict, so the node must hold the instance's whole memory plus fork headroom, or the instance can be OOM-killed while the other node has free memory.

Where else does a VM add latency: interrupts, steal time, and noisy neighbours?

The latency guide puts it plainly: "System induced latencies are significantly higher on a virtualized environment than on a physical machine." It reports measuring up to 40 ms on systems that otherwise looked normal.

Interrupt placement. Network interrupts that land on the core running Valkey's main thread compete with it. Valkey's 1 million RPS reproduction guide pins the NIC's interrupt lines to chosen cores and moves the main thread to a core that handles none. Its commands, for an interface named eth0:

grep eth0 /proc/interrupts | awk '{print $1}' | cut -d : -f 1   # IRQ numbers
for i in {48..51}; do echo 1000 > /proc/irq/$i/smp_affinity; done
sudo taskset -cp 3 `pidof valkey-server`

Interpretation: the IRQ numbers 48 to 51 and the mask 1000 (CPU 12) come from that guide's host; read your own from /proc/interrupts. If irqbalance runs, check its configuration, because it manages the same masks. Most workloads never need this.

Steal time and noisy neighbours. The st column in vmstat is "time stolen from a virtual machine" and shows up as a higher intrinsic-latency maximum and a wider p99. Burstable instances add a second effect: AWS describes the T family as giving a baseline CPU performance with the ability to burst, so a latency floor measured during a burst may not hold later. Prefer fixed-performance families for latency-sensitive primaries.

How do you measure the latency floor with --intrinsic-latency?

valkey-cli --intrinsic-latency <seconds> times a tiny, fixed piece of work over and over and prints each new maximum it sees. It does not connect to the server, so run it on the Valkey host. The latency guide calls the result intrinsic latency: the floor the kernel, hypervisor, and hardware set before Valkey adds anything.

# Run on the Valkey host, during representative load, for 100 seconds
valkey-cli --intrinsic-latency 100

Interpretation: the output is a series of Max latency so far: N microseconds. lines, then the number of runs, the average per run, and how many times longer the worst run took. The guide's quiet example tops out at 115 microseconds, and its noisy virtualized example reaches 9,671 microseconds. Valkey's worst case on that host cannot be better than this figure, so client tail latency in that range points at the host, not at Valkey.

Run it before go-live and after any host change, and keep the results. In an incident, a fresh run answers the first question: did the floor move? If it did, look at the host. If not, work through the sibling Valkey latency incident runbook, which covers COMMANDLOG, LATENCY DOCTOR, and the in-server events.

What belongs on a cloud-instance checklist for Valkey?

The checklist below puts the earlier sections in the order you would run them on a new host. Each step is a read or a reversible sysfs or sysctl write, except the restart.

For packages and container images, confirm the Valkey version they ship. In Kubernetes, THP, overcommit, and clocksource are node settings, so the checklist runs on the node image.

Frequently asked questions

Key takeaways

Set a Valkey latency baseline with JusDB

JusDB's Valkey performance tuning team can review your instance types, clocksource, THP, swap, and NUMA settings against the 9.1 release notes, check which nodes still run builds with the clock bugs, and help you record an intrinsic-latency baseline per host.

Bring your Valkey versions, instance families, kernel and image details, and current p99 targets. Your team approves every change on production hosts.

Related reading

• Valkey performance tuning: memory, persistence, and cluster configuration
• Valkey 9.1: database ACLs, Lua, and I/O threading
• Redis performance tuning for high-traffic applications
• Valkey documentation: Diagnosing latency issues