Finding
Burstable pod limits and missing preStop hooks trigger CLUSTERDOWN during EKS node rotation
The StatefulSet configured maxmemory 30gb inside a 32Gi container limit, causing OOMKilled (Exit 137) whenever BGSAVE forked. Furthermore, Karpenter evicted pods without running CLUSTER FAILOVER first.
1. Hardened Valkey 8.0 StatefulSet: Pinned QoS, Karpenter lock & graceful preStop failover
valkey-statefulset.yaml · production k8s spec
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: valkey-cluster
namespace: data-prod
spec:
serviceName: valkey-headless
replicas: 6
podManagementPolicy: Parallel
template:
metadata:
annotations:
karpenter.sh/do-not-disrupt: "true"
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/name: valkey-cluster
containers:
- name: valkey
image: valkey/valkey:8.0-alpine
args:
- "valkey-server"
- "/etc/valkey/valkey.conf"
- "--maxmemory"
- "22gb"
- "--maxmemory-policy"
- "allkeys-lfu"
- "--io-threads"
- "4"
- "--io-threads-do-reads"
- "yes"
- "--cluster-preferred-endpoint-type"
- "hostname"
resources:
requests:
cpu: "4"
memory: "32Gi"
limits:
memory: "32Gi"
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "/scripts/graceful-cluster-failover.sh"]2. PodDisruptionBudget & cgroup memory verification
valkey-pdb.yaml & kubectl verification
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: valkey-cluster-pdb
namespace: data-prod
spec:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/name: valkey-cluster
---
# Verify Pinned QoS class and cluster hostname endpoints
# kubectl get pod valkey-cluster-0 -n data-prod -o jsonpath='{.status.qosClass}'
# kubectl exec -it valkey-cluster-0 -n data-prod -- valkey-cli cluster slots






