Free audit

View Audit Scope
JusDB24x7 P1 Incident Bridge Available · Valkey 7.2 & 8.0 · Sentinel & Cluster

Production Valkey support for OOM lockouts, replication storms, and failover.

JusDB provides production Valkey incident support for engineering teams running mission-critical caches, session stores, and queues. When a primary rejects writes with OOM command not allowed, replicas enter endless FULLRESYNC fork loops, Sentinel loses quorum, or a 16,384-slot Cluster flips to cluster_state:fail, a senior DBRE joins your bridge within 15 minutes for P1 incidents under a 24x7 support agreement.
The first 30-minute call covers your databases, current ownership, and required coverage. No production access is needed. Dealing with an active Valkey outage right now? Escalate a live incident.
  • NDA before access
  • Read-only by default
  • Written scope before work starts

valkey_incident_bridge

P1 Incident Bridge

Senior Valkey DBRE under a 24x7 support agreement

Direct escalation to senior Valkey DBREs - online CONFIG SET recovery without cold cache wipes.

P1 Initial Response
15 minutes
For critical production outages under a 24x7 support agreement
Business-Hours Target
2 to 4 hours
During covered business hours
Topologies Covered
Sentinel & Cluster
Self-hosted Valkey 7.2/8.0, ElastiCache, Memorystore, and Kubernetes
Post-Incident Output
Written RCA & Diff
Permanent valkey.conf patch, CONFIG REWRITE, and alert thresholds

Selected clients

  • Innoviti
  • Blitz
  • Freecharge
  • Nango
  • Recochain
  • Screen Facts
  • Nexora
  • WeCredit

Production emergencies we resolve

Six high-severity Valkey incidents we resolve on the bridge.

Restarting an in-memory datastore during an incident wipes warm caches and triggers a thundering herd against your primary SQL or NoSQL database. We stabilize Valkey online first.

OOM_COMMAND_NOT_ALLOWED

Valkey rejects all writes with OOM command not allowed when used memory > 'maxmemory'

What is usually happening
Namespace filled maxmemory with noeviction policy, or volatile-lru/ttl eviction found zero keys with TTLs while traffic surged.
Where we start
Inspect INFO memory and INFO keyspace, identify un-TTL'd key prefixes or bigkeys via non-blocking SCAN, and apply approved online CONFIG SET eviction or buffer adjustments.
REPLICA_FULL_RESYNC_STORM

Replicas repeatedly disconnect and trigger full RDB resynchronizations, spiking primary RAM and NIC

What is usually happening
Write bursts overflow client-output-buffer-limit replica (or repl-backlog-size), dropping the replica link and triggering a fresh BGSAVE fork on every reconnect.
Where we start
Raise client-output-buffer-limit replica and repl-backlog-size dynamically via CONFIG SET, enable diskless sync pacing, and break the fork loop.
SENTINEL_QUORUM_OR_SPLIT_BRAIN

Primary node died during an AZ blip, but Sentinel failed to promote a replica (or two primaries exist)

What is usually happening
Sentinel achieved +sdown (subjective down) without +odown quorum, or an isolated former primary continued accepting writes without min-replicas-to-write.
Where we start
Inspect SENTINEL master and SENTINEL replicas state, execute controlled SENTINEL FAILOVER or REPLICAOF reconciliation, and salvage any diverged AOF/RDB keys.
CLUSTER_STATE_FAIL_SLOTS

Valkey Cluster enters cluster_state:fail and rejects traffic across all shards

What is usually happening
A shard lost both its primary and replica (or a slot migration stalled in MIGRATING/IMPORTING state) with cluster-require-full-coverage set to yes.
Where we start
Check valkey-cli --cluster check output, fix stuck migrating slots with --cluster fix, promote surviving replicas via CLUSTER FAILOVER TAKEOVER, and restore slot coverage.
MAIN_THREAD_BLOCKED_LUA_SLOWLOG

Valkey CPU hits 100% on a single core while all client commands time out

What is usually happening
A runaway Lua script, accidental KEYS * / SMEMBERS on a 10M-item set, or synchronous DEL on a massive bigkey blocked the single command execution thread.
Where we start
Inspect SLOWLOG GET and CLIENT LIST, terminate blocking scripts with SCRIPT KILL / FUNCTION KILL or CLIENT KILL, and enable lazyfree-lazy-server-del.
AOF_FSYNC_OR_DISK_FULL_LOCK

Valkey logs Asynchronous AOF fsync is taking too long and blocks writes with MISCONF errors

What is usually happening
Underlying EBS/PD disk IOPS saturated during an AOF rewrite (with no-appendfsync-on-rewrite set to no) or persistent volume reached 100% capacity.
Where we start
Set no-appendfsync-on-rewrite yes and stop-writes-on-bgsave-error dynamically while clearing disk pressure, then trigger a controlled BGREWRITEAOF.

What Valkey support covers

Six incident-response and stabilization workstreams.

Built for teams that run Valkey day to day in-house and want senior DBREs on call when production memory, replication, or cluster state breaks.

01

P1 Outage Triage & Online Stabilization

Direct escalation to senior Valkey DBREs when your fleet hits OOM write lockouts, p99 latency cliffs, or main-thread stalls.

  • Live inspection of INFO all, SLOWLOG, LATENCY DOCTOR, and MEMORY STATS
  • Zero-restart stabilization using runtime CONFIG SET and non-blocking UNLINK
CONFIG SET · LATENCY DOCTOR · SLOWLOG GET
02

Sentinel Quorum & Failover Recovery

Restoring write availability when Sentinel quorums stall, replicas refuse promotion, or network partitions create dual primaries.

  • Manual or Sentinel-orchestrated promotion with client VIP/DNS verification
  • Post-partition split-brain reconciliation and min-replicas-to-write hardening
SENTINEL FAILOVER · REPLICAOF · ckquorum
Valkey high availability
03

Valkey Cluster Slot & Split-Brain Rescue

Emergency recovery for 16,384-slot Valkey Cluster deployments stuck in cluster_state:fail or mid-resharding slot locks.

  • Resolving stuck MIGRATING / IMPORTING hash slots via valkey-cli --cluster fix
  • Controlled CLUSTER FAILOVER FORCE / TAKEOVER when master shards are lost
valkey-cli --cluster fix · CLUSTER FAILOVER
Valkey Cluster services
04

Replication Storm & Buffer Overflow Mitigation

Breaking cascading full-resync loops where primary fork() overhead and replica output buffers starve application traffic.

  • Dynamic tuning of client-output-buffer-limit replica and repl-backlog-size
  • Paced replica re-attachment and Valkey 8.0 dual-channel replication tuning
INFO replication · client-output-buffer-limit · PSYNC2
05

RDB / AOF Corruption & Persistence Recovery

Recovering data and clearing MISCONF write blocks when disks fill up or AOF/RDB files corrupt after an unclean host crash.

  • Repairing truncated append-only files using valkey-check-aof and valkey-check-rdb
  • Online relief of AOF fsync stalls via no-appendfsync-on-rewrite
valkey-check-aof · valkey-check-rdb · BGREWRITEAOF
06

Root-Cause Postmortem & Config Persistence

Every incident closes with a written timeline, CONFIG REWRITE persistence check, and Prometheus exporter alert rules.

  • Ensuring runtime CONFIG SET fixes are persisted via CONFIG REWRITE / GitOps
  • Alert threshold tuning for memory fragmentation, replication lag, and slowlog rates
CONFIG REWRITE · redis_exporter · RCA Dossier
Valkey performance tuning

Shared responsibility & P1-P4 response targets

Clear incident response targets and ownership boundaries.

We publish exact initial-response targets by severity tier and define what your cloud provider covers versus what JusDB resolves inside the Valkey engine.

Initial-response targets by severity

P1 to P4 escalation matrix

Initial-response targets are 2 to 4 hours during covered business hours, Mon–Fri 09:00–18:00 IST (UTC+5:30). Where your service agreement includes 24x7 on-call, the initial-response target for P1 incidents is 15 minutes. Targets are not resolution guarantees. Our engineering team is in India; business-hours coverage is Mon–Fri 09:00–18:00 IST (UTC+5:30), and 24x7 on-call is available under your service agreement.

SeverityValkey Condition24x7 TargetBusiness Hours
P1 · CriticalProduction database down, primary unreachable, data corruption risk, or active failover event.15 minutes2 to 4 hours
P2 · HighSevere latency spike, replication lag climbing rapidly, connection exhaustion, or degraded HA node.30 minutes2 to 4 hours
P3 · MediumNon-critical slow query regression, staging issue, backup warning, or replication check alert.60 minutes2 to 4 hours
P4 · LowConfiguration question, index advisory, planned schema review, or general operational request.90 minutes2 to 4 hours
01 · Page
Pager / Slack / ticket triggers on-call DBRE
02 · Triage
Read-only INFO, SLOWLOG & LATENCY check
03 · Mitigate
Online CONFIG SET & quorum restoration
04 · Review
CONFIG REWRITE, RCA & alert rules

Initial-response targets, not resolution guarantees. The target for your engagement is set in your service agreement.

Your team retains

Customer ownership boundaries

  • Set business priorities, application requirements, and release plans
  • Approve production changes, except routine actions you have pre-approved
  • Own your cloud accounts, access policies, and vendor relationships
  • Decide when maintenance can happen and how quickly service must recover
  • Move the work in-house or to another provider after your agreement's notice period, with your data returned or deleted and the transition help your agreement sets out (see Terms, section 8)
JusDB support owns

Engineering accountability

  • Monitor database health, investigate alerts, and involve the right people
  • Review query performance, replication, capacity, and backup health
  • Carry out approved changes and update the operating instructions they affect
  • Report completed work, open risks, and recommended next steps
VM, EBS/PD volume & ElastiCache hardware
Cloud provider replaces failed underlying VMs.
Your infra team manages VMs/K8s nodes; JusDB coordinates failover and replica promotion.
OOM lockouts, eviction & bigkey stalls
Cloud support does not touch your keys or eviction rules.
JusDB diagnoses memory spikes, unlinks offending keys, and tunes maxmemory-policy live.
Replication desync & Sentinel quorum loss
Abstracted in managed tiers; manual in VM/K8s fleets.
JusDB breaks full-resync loops, fixes Sentinel quorum, and reconciles split-brain state.
Valkey Cluster slot failure & resharding locks
Requires operator intervention when slots stick in MIGRATING.
JusDB runs --cluster fix, restores 16,384-slot coverage, and stabilizes client routing.
Post-incident RCA & CONFIG REWRITE
Cloud ticket closes once host is reachable.
JusDB delivers a written postmortem with valkey.conf diffs and Prometheus alert rules.

Incident lifecycle simulator

How we break a FULLRESYNC fork loop and OOM lockout online.

Step through a real-world Valkey P1 outage to see how runtime CONFIG SET adjustments restore replication and writes without flushing your cache.

valkey_p1_incident_simulator

Application and routing

App Pool -> Valkey Primary (p99 < 0.5ms) -> PSYNC2 Streaming to Replicas

Active
  • valkey-primary-01

    Master · Single-Thread Nominal

    used_memory=62% · frag=1.05 · blocked_clients=0

  • valkey-replica-01 & 02

    Connected Replicas

    master_link_status=up · lag=0s · RDB backup clean

  • Sentinel / Cluster Bus

    Quorum & Slots Healthy

    cluster_state=ok · 16,384 slots covered · 0 slowlogs

Memory sits comfortably below maxmemory, replicas stream cleanly over PSYNC2, and command execution completes in sub-millisecond time.

Production incident runbook

Live valkey-cli triage and online stabilization commands.

During an incident bridge, we inspect memory, replication buffers, and latency events read-only first, then apply targeted runtime fixes.

Post-Incident RCA #08 · P1 Recovery

Cascading replica disconnect and CoW memory exhaustion during flash-sale traffic

A write surge exceeded the default client-output-buffer-limit replica 256mb 64mb 60 threshold. Both replicas disconnected simultaneously and requested FULLRESYNC, triggering a BGSAVE fork on the primary whose copy-on-write pages pushed memory above maxmemory.

Immediate mitigation: Expanded client-output-buffer-limit replica to 1024mb 512mb 120 and repl-backlog-size to 512mb via CONFIG SET, breaking the full-resync loop immediately.
Permanent fix: Persisted via CONFIG REWRITE and upgraded to Valkey 8.0 dual-channel replication.
Stabilized online without flushing cache or restarting primary

01 · Read-only incident triage (OOM, replication links & slowlog)

# 1. Inspect memory pressure, CoW fork size, and eviction counters
valkey-cli INFO memory
valkey-cli INFO stats | grep -E "evicted_keys|rejected_connections|sync_full|sync_partial_err"

# 2. Check replica link status, offset lag, and output buffer usage
valkey-cli INFO replication
valkey-cli CLIENT LIST TYPE replica

# 3. Inspect latency spikes and slow commands blocking the event loop
valkey-cli LATENCY DOCTOR
valkey-cli SLOWLOG GET 15

02 · Online mitigation via dynamic CONFIG SET & CONFIG REWRITE

# 1. Expand replica output buffer and backlog online to stop FULLRESYNC loops
valkey-cli CONFIG SET client-output-buffer-limit "normal 0 0 0 replica 1073741824 536870912 120 pubsub 33554432 8388608 60"
valkey-cli CONFIG SET repl-backlog-size 536870912

# 2. Enable non-blocking lazy-free deletion and active jemalloc defragmentation
valkey-cli CONFIG SET lazyfree-lazy-eviction yes
valkey-cli CONFIG SET lazyfree-lazy-expire yes
valkey-cli CONFIG SET activedefrag yes

# 3. Persist runtime fixes to valkey.conf once cluster stabilizes
valkey-cli CONFIG REWRITE
Evaluation Vector
JusDB DBRE
In-House DBALegacy AgencyDeveloper Generalist
Initial-Response Targets & Escalation HierarchyA dedicated Slack/Teams war room with named DBREs for Sev-1 incidents. Initial-response targets: 2 to 4 hours in covered business hours; 15 minutes for P1 incidents with 24x7 on-call. Targets, not resolution guarantees.Best-effort notification dependent on individual engineer alert settings, personal sleep schedules, and solo on-call availability.4–8 hour response window with tickets routed through non-technical tier-1 triage operators before engineer dispatch.No incident SLA; software engineers pulled away from active feature sprints to debug database outages under high pressure.
Low-Level Valkey C-Core & Kernel Network DepthDeep mastery of Valkey C-core internals, multi-threaded I/O (io-threads), jemalloc memory arena allocation, Linux kernel somaxconn, and TCP backlog tuning.Familiar with standard valkey-cli commands and basic key-value operations, but lacks deep C-core engine profiling or Linux kernel socket buffer expertise.Treats Valkey as identical to legacy single-threaded Redis, misconfiguring multi-threading, active defragmentation, and jemalloc parameters.Basic SDK/client API usage without understanding in-memory serialization overhead, fork copy-on-write memory bloat, or thread contention.
Emergency War Room Collaboration & Direct DBRE DispatchA dedicated bridge via Slack, Teams, or Zoom with live terminal screen sharing, joint triage debugging, and regular incident situation updates.Solo engineer isolated on emergency bridge, navigating high-stress production outages without senior peer review or escalation channels.Rigid ticket-only updates with asynchronous replies every several hours, strictly refusing to join live incident debugging bridges.Trial-and-error commands executed in panic during outages, risking catastrophic FLUSHALL, replica desync, or node drop.
Proactive Root Cause Analysis (RCA) & PostmortemsBlameless engineering RCA on the timeline set in your service agreement, including valkey-cli telemetry timelines, latency spikes, and recommended maxmemory fixes.Superficial postmortems frequently abandoned or delayed due to urgent product development backlogs and competing roadmap priorities.Generic closure notes stating 'daemon restarted, cluster green' without identifying underlying memory leaks, large keys, or replication drop causes.Postmortem skipped as soon as the node restarts, leaving the cluster vulnerable to repeat memory crashes and eviction storms.
Diagnostic Telemetry & Tooling (valkey-cli, latency, slowlog)Deep diagnostic mastery using valkey-cli info memory, latency doctor, slowlog, MEMORY USAGE profiling, and real-time jemalloc fragmentation tracking.Standard OS-level CPU/RAM metrics missing jemalloc allocator fragmentation ratios, replication backlog buffers, and client output buffer drops.Basic ping and uptime monitoring with zero visibility into Valkey slowlog, blocked clients, or Sentinel quorum flapping.Application APM metrics only; lacks database-level telemetry instrumentation or awareness of single-threaded command blocking.
Security Hardening & Compliance (ISO 27001 / SOC 2)Zero-trust ephemeral access via audited bastions, mutual TLS (mTLS) inter-node encryption, fine-grained Valkey ACL rules, and practices aligned with SOC 2 Type II and ISO 27001 (certification in progress).Static credentials, default master auth passwords, and shared administrative keys stored across team credential vaults without per-session recording.Third-party contractors accessing production Valkey nodes over unmonitored shared VPNs without tamper-proof audit trails.Valkey ports (6379/16379) exposed across internal subnets with disabled protected-mode or permissive default user permissions.

Support & operational models

Choose how our Valkey engineers back your team.

Retain JusDB for reactive P1-P4 incident escalation or hand off routine operations under a Remote DBA retainer. A monthly retainer for a set number of engineering hours, or a monthly fee per database instance, HA cluster, or region; projects are scoped and quoted separately.

Reactive Escalation

Production Support Retainer

Your team runs day-to-day ops; JusDB provides P1-P4 incident response, OOM and split-brain recovery, and written postmortems (24x7 on-call available under a 24x7 support agreement).

Continuous Operations

Valkey Remote DBA

We own continuous telemetry monitoring, rolling upgrades, backup verification, and capacity forecasting so incidents are caught early.

Architecture Review

Valkey Consulting

Fixed-scope audit of BSD-3 migration readiness, Sentinel vs Cluster sizing, and Valkey 8.0 performance optimization.

Procurement and security

Ready for legal and security review.

Mutual NDA before any access, least-privilege credentials in your vault, and security controls aligned with ISO 27001 and SOC 2 Type II.
  • Mutual NDA before any access
  • Named individual accounts only
  • Read-only access by default
  • ISO 27001 & SOC 2 Type II: aligned, certification in progress
  • Mutual NDA before any access

    We sign a mutual NDA before viewing schemas, logs, slow-query captures, or cloud topology.

  • Least-privilege, named credentials

    Engineers connect through your bastion, VPN, or SSM using named accounts with read-only permissions by default.

  • Change control & approval gates

    Production DDL, parameter changes, and failovers follow your change-management workflow with rollback steps prepared first.

  • Named leadership & direct escalation

    Engagements are led by named senior database reliability engineers in India with direct Slack, Teams, and PagerDuty channels.

Valkey support FAQ

Incident response, SLAs, and scope questions.

How our Valkey support bridge operates, how P1 targets work, and how we stabilize live production caches.

Still have a question?

Ask it on a scoping call. No production access needed.

Book a 30-min scoping call (opens Google Calendar in a new tab)

How is Valkey Support different from Valkey Remote DBA?

Valkey Support is reactive incident response: your engineering team operates your Valkey fleet day to day, and you page JusDB when an emergency hits (OOM write lockouts, replication desync loops, Sentinel quorum loss, or Cluster split-brain). Valkey Remote DBA (/databases/valkey/remote-dba) is continuous managed operations where our team monitors your fleet daily and executes routine maintenance before alerts become outages.

What response targets apply to Valkey support incidents?

Initial-response targets are 2 to 4 hours during covered business hours, Mon–Fri 09:00–18:00 IST (UTC+5:30). Where your service agreement includes 24x7 on-call, the initial-response target for P1 incidents is 15 minutes. Targets are not resolution guarantees. Our engineering team is in India; business-hours coverage is Mon–Fri 09:00–18:00 IST (UTC+5:30), and 24x7 on-call is available under your service agreement. Acknowledgment means a senior Valkey engineer is on the bridge inspecting INFO, SLOWLOG, and LATENCY DOCTOR output, not an automated ticket reply.

Can you resolve Valkey OOM or replication outages without restarting the server?

In the vast majority of cases, yes. Because Valkey supports runtime reconfiguration via CONFIG SET, we can adjust maxmemory, maxmemory-policy, client-output-buffer-limit, repl-backlog-size, activedefrag, and AOF fsync policies online without restarting the daemon and losing your warm in-memory cache.

What kinds of Valkey and Redis incidents do you handle?

We handle OOM write rejections, copy-on-write fork crashes during BGSAVE/BGREWRITEAOF, endless replica full-resync loops, Sentinel quorum failures, Valkey Cluster slot coverage failures (cluster_state:fail), main-thread stalls from slow Lua scripts or O(N) commands, jemalloc memory fragmentation spikes, and AOF/RDB corruption.

How does JusDB support interact with AWS ElastiCache or GCP Memorystore support?

Cloud provider support covers the underlying virtual machine and control-plane API. They do not debug why your Valkey engine is evicting keys, why a Lua script blocked the event loop, why replication buffers are overflowing, or why Cluster slots are skewed. We own the engine-level triage and coordinate with cloud support when a hardware replacement is needed.

Do we need a support contract in place before an incident happens?

We can take on cold-start emergency incidents once mutual NDA, scope, and access are agreed. However, having a support agreement in place ahead of time means read-only access and topology runbooks are already verified, allowing us to hit our 15 minutes P1 target under a 24x7 support agreement. A monthly retainer for a set number of engineering hours, or a monthly fee per database instance, HA cluster, or region; projects are scoped and quoted separately.

Valkey engineering ecosystem

Pair reactive support with proactive Valkey engineering.

Explore our adjacent Valkey services for ongoing fleet management, Cluster sharding, and Redis-to-Valkey migrations.

Operations

Valkey Remote DBA

Continuous day-to-day fleet monitoring, rolling upgrades, and proactive capacity management.

Explore service
Architecture

Valkey Consulting

BSD-3 migration strategy, Sentinel vs Cluster sizing, and Valkey 8.0 architecture reviews.

Explore service
Sharding

Valkey Cluster Services

16,384 hash-slot sharding, online slot rebalancing, and MOVED/ASK redirection tuning.

Explore service
Resilience

Valkey High Availability

Sentinel quorum design, cross-AZ replica placement, and split-brain prevention.

Explore service
Optimization

Valkey Performance Tuning

Sub-millisecond latency optimization, jemalloc activedefrag, and bigkey remediation.

Explore service
Migration

Redis to Valkey Migration

Low-downtime REPLICAOF live cutovers and ElastiCache/Memorystore conversions.

Explore service

Related Valkey Services

Explore more ways our Valkey experts can help with your database infrastructure.

Put senior Valkey DBREs on your incident bridge

Stop debugging OOM lockouts and Sentinel split-brain alone at 2 AM.

Establish a production support agreement with pre-verified read-only access and topology runbooks, or request a free cluster assessment to catch hidden memory risks.