Finding
40 GiB JVM heap broke compressed oops and caused cgroup OOM kills on EKS node rotations
Data pods were deployed in a single 48 GiB container with -Xmx40g and co-located master roles. Crossing the 32 GiB compressed-oops boundary reduced effective addressable heap capacity while leaving only 8 GiB for Lucene mmap segments, triggering OOMKilled (ExitCode 137) during heavy merges.
1. ECK CRD: Dedicated Master and Data-Hot nodeSets with 30 GiB heap and AZ spread
elasticsearch-prod.yaml · ECK v1 CRD
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: prod-search
namespace: elastic-system
spec:
version: 8.15.3
podDisruptionBudget:
spec:
maxUnavailable: 1
nodeSets:
- name: master
count: 3
config:
node.roles: ["master"]
podTemplate:
spec:
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: "-Xms4g -Xmx4g"
resources:
requests: { memory: 8Gi, cpu: "2" }
limits: { memory: 8Gi, cpu: "2" }
- name: data-hot
count: 6
config:
node.roles: ["data_hot", "data_content", "ingest"]
cluster.routing.allocation.awareness.attributes: k8s_node_name,zone
node.attr.k8s_node_name: ${HOSTNAME}
podTemplate:
spec:
initContainers:
- name: sysctl
securityContext: { privileged: true }
command: ["sh", "-c", "sysctl -w vm.max_map_count=262144"]
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: "-Xms30g -Xmx30g -XX:+UseG1GC"
resources:
requests: { memory: 62Gi, cpu: "8" }
limits: { memory: 62Gi, cpu: "8" }2. Verify compressed oops and delayed unassigned shard timeout
elasticsearch api · verification & drain safety
# Confirm compressed object pointers are enabled ("using_compressed_ordinary_object_pointers": "true")
GET _nodes/jvm?filter_path=nodes.*.jvm.mem.heap_max_in_bytes,nodes.*.jvm.using_compressed_ordinary_object_pointers
# Prevent shard relocation storms during brief Kubernetes pod restarts
PUT /_all/_settings
{
"settings": {
"index.unassigned.node_left.delayed_timeout": "5m"
}
}






