Free audit

View Audit Scope
OpenSearch on K8sOpenSearch Operator, Hot-Warm nodePools & EKS/GKE ops

OpenSearch on Kubernetes: Operator & Cloud-Native Ops

JusDB deploys and operates production OpenSearch on Kubernetes across Amazon EKS, Google GKE, Azure AKS, OpenShift, and bare-metal clusters. Search infrastructure engineers configure the official OpenSearch Kubernetes Operator (OpenSearchCluster CRD) and Helm charts, separate ClusterManager, Hot Data, Warm Data, and k-NN nodePools, tune JVM heap and off-heap cgroup limits, and automate S3 snapshots and rolling upgrades under your service agreement.

Book a 30-min scoping call (opens Google Calendar in a new tab)See K8s architecture

The first 30-minute call covers your databases, current ownership, and required coverage. No production access is needed. Or request a free audit of one database instance.

  • NDA before access
  • GitOps-ready Operator CRDs
  • Written scope before work starts

OpenSearch Kubernetes architecture

Select a Kubernetes component

Official OpenSearch Kubernetes Operator & multi-tier nodePools

OpenSearchCluster CRD

Declarative lifecycle orchestration for ClusterManager, Data (Hot/Warm/Cold/k-NN), Ingest, and Dashboards StatefulSets with shard-allocation-aware rolling upgrades.

Inspect K8s operator states

Aligned with the OpenSearch Kubernetes Operator specification.

Kubernetes platforms
Amazon EKS, Google GKE, Azure AKS, Red Hat OpenShift, and bare-metal Kubernetes with local NVMe CSI
Initial-response targets
15 minutes for a critical (P1) incident under a 24x7 support agreement; 2 to 4 hours in business hours. See details
Coverage hours
Business hours (03:30–12:30 UTC) by default; 24x7 on-call available under a 24x7 support agreement

Selected clients

  • Innoviti
  • Blitz
  • Freecharge
  • Nango
  • Recochain
  • Screen Facts
  • Nexora
  • WeCredit

When teams call us

Common OpenSearch on Kubernetes failures we fix

We fix vm.max_map_count startup crashes, Exit Code 137 cgroup OOMKills, node-drain shard rebalance storms, ClusterManager split-brain, TLS securityadmin lockouts, and S3 IRSA snapshot errors.

Common OpenSearch on Kubernetes production symptoms and how JusDB resolves them
SymptomWhat is usually happeningWhere we start
MAX_MAP_COUNT_BOOTSTRAPOpenSearch pods crash at startup with max virtual memory areas [65530] is too lowLucene mmapfs requires vm.max_map_count >= 262144 on the underlying Kubernetes worker node kernel.Configure node-level sysctl via DaemonSet, EKS launch template user-data, or the OpenSearch Operator setVMMaxMapCount initContainer.
K8S_CGROUP_OOM_137Data or k-NN pods terminate with Exit Code 137 (OOMKilled)JVM -Xmx is set above 50% of the pod memory limit, leaving no cgroup headroom for Lucene mmap page cache, Netty direct buffers, or FAISS k-NN native memory.Set Guaranteed QoS (requests == limits = 64Gi), cap -Xms/-Xmx at 31g, and configure knn.memory.circuit_breaker.limit.
NODE_DRAIN_SHARD_STORMKubernetes cluster upgrade triggers hours of YELLOW state and disk I/O saturationKubelet drains a data pod without setting cluster.routing.allocation.enable: primaries or delayed_timeout, causing immediate network shard copying.Use the OpenSearch Kubernetes Operator drain controller, set index.unassigned.node_left.delayed_timeout: 5m, and enforce PodDisruptionBudgets.
SPLIT_BRAIN_QUORUM_LOSSClusterManager pods lose quorum during AZ maintenanceTwo of 3 cluster_manager pods were scheduled onto the same availability zone or worker node without hard anti-affinity.Enforce requiredDuringSchedulingIgnoredDuringExecution anti-affinity across topology.kubernetes.io/zone for the cluster_manager nodePool.
SECURITY_ADMIN_CERT_LOCKRolling TLS certificate renewal locks out securityadmin.sh and DashboardsTransport and HTTP certificates in Kubernetes Secrets were rotated without matching DN/SAN entries in opensearch.yml or running securityadmin.Automate TLS lifecycle with cert-manager and the OpenSearch Operator securityConfig reconciliation loop.
S3_IRSA_SNAPSHOT_FAILRepository-s3 snapshot registration fails with AccessDenied on EKS/GKEData pods lack IAM Roles for Service Accounts (IRSA / Pod Identity) or GKE Workload Identity bindings and opensearch-keystore reload.Bind an IAM-annotated ServiceAccount to the OpenSearchCluster CRD and configure repository-s3 with regional STS endpoints.

Response targets and escalation

How fast does JusDB respond to an OpenSearch Kubernetes incident?

Initial-response targets are 2 to 4 hours during covered business hours, Mon–Fri 09:00–18:00 IST (UTC+5:30). Where your service agreement includes 24x7 on-call, the initial-response target for P1 incidents is 15 minutes. Targets are not resolution guarantees. Standard business hours run 03:30–12:30 UTC.

Published initial-response targets by severity

  • 2 to 4 hours

    Business hours

    During covered business hours

  • 15 minutes

    24x7 on-call

    P1 incidents

SeverityInitial responseDefinitionChannel & coverage
P1 · Critical15 minutesProduction database down, primary unreachable, data corruption risk, or active failover event.24x7 on-call (under your service agreement) · Direct PagerDuty, Opsgenie, or phone page plus incident bridge
P2 · High30 minutesSevere latency spike, replication lag climbing rapidly, connection exhaustion, or degraded HA node.24x7 on-call (under your service agreement) · On-call alert plus shared Slack or Teams channel
P3 · Medium60 minutesNon-critical slow query regression, staging issue, backup warning, or replication check alert.24x7 on-call (under your service agreement) · Shared Slack or Teams channel or support ticket
P4 · Low90 minutesConfiguration question, index advisory, planned schema review, or general operational request.24x7 on-call (under your service agreement) · Shared Slack or Teams channel or support ticket

Our engineering team is in India; business-hours coverage is Mon–Fri 09:00–18:00 IST (UTC+5:30), and 24x7 on-call is available under your service agreement.

Initial-response targets, not resolution guarantees. The target for your engagement is set in your service agreement.

What happens when a P1 Kubernetes incident pages

  1. Page

    Direct alert to the on-call OpenSearch & Kubernetes engineer

    No tier-1 queue; immediate triage under your service agreement.

  2. Inspect

    Read-only OpenSearchCluster CRD, pod, JVM, and shard diagnostics

    Correlate kubelet OOMs, allocation explain output, and Operator logs.

  3. Stabilize

    Restore GREEN status, freeze rebalance storms, and cap memory

    Reroute unassigned primaries, clear read-only locks, or adjust heap ratios.

  4. Harden

    Written post-incident review and GitOps manifest pull request

    Permanent CRD, PDB, and ISM fixes committed to your repository.

Scope of Kubernetes engineering

What does OpenSearch on Kubernetes consulting & operations cover?

We deliver hardened OpenSearchCluster CRDs, multi-tier Hot/Warm/k-NN nodePools, 50% JVM cgroup tuning, cert-manager TLS, IRSA S3 snapshots, and smart rolling upgrades.

  • OpenSearch Operator & nodePool design

    We author production OpenSearchCluster CRDs and Helm values with dedicated nodePools for ClusterManager, Hot Data, Warm Data, Ingest, and k-NN.

    • 3-AZ ClusterManager quorum with strict anti-affinity and PodDisruptionBudgets
    • Shard allocation awareness mapped automatically to Kubernetes topology zones

    opensearch-k8s-operator · OpenSearchCluster CRD · Helm / ArgoCD

    Explore OpenSearch consulting

  • JVM, cgroup v2 & k-NN memory engineering

    Eliminate Exit Code 137 OOMKills by balancing JVM heap (max 31 GiB), Linux page cache, and native FAISS/NMSLIB off-heap vector memory.

    • Guaranteed QoS pod specs (64 GiB pod limit paired with 31 GiB G1GC JVM heap)
    • Kernel sysctl automation (vm.max_map_count = 262144, swapoff) without privileged pods

    Guaranteed QoS · G1GC · vm.max_map_count · k-NN Breaker

  • NVMe + EBS/PD-SSD tiered storage classes

    Pair local NVMe CSI or high-IOPS EBS GP3 volumes on Hot nodePools with dense Warm storage and S3 Searchable Snapshots.

    • ISM policies routing indices across Hot and Warm nodePool roles
    • Online PVC volume expansion and disk watermark protection

    Local NVMe CSI · EBS GP3 · ISM Node Attributes · PVC Expansion

    Read the shard sizing & ILM/ISM guide

  • cert-manager TLS & Security Plugin GitOps

    Automate transport and HTTP mTLS certificate rotation, SAML/OIDC single sign-on, and Fine-Grained Access Control (FGAC) in Kubernetes.

    • Zero-downtime TLS rotation via cert-manager and Operator hot reload
    • Declarative internal_users, roles, and role_mappings managed via Kubernetes Secrets

    cert-manager · Security Plugin · FGAC · SAML / OIDC

  • IRSA S3 snapshots & disaster recovery

    Configure credential-free S3, GCS, or Azure Blob snapshot repositories using Kubernetes Workload Identity and scheduled SM policies.

    • AWS IRSA / EKS Pod Identity and GKE Workload Identity for repository-s3
    • Automated staging restore verification and Cross-Cluster Replication (CCR)

    repository-s3 · IRSA / Workload Identity · Snapshot Management

  • Smart rolling upgrades & 24x7 support

    Upgrade OpenSearch versions safely with Operator-managed shard allocation freezes and hand over day-2 monitoring under your agreement.

    • Pre-restart _flush/synced and allocation disable/enable orchestration
    • 24x7 P1 incident response available under a 24x7 support agreement

    Rolling Upgrade · Prometheus Exporter · Grafana · PagerDuty

    Explore OpenSearch support services

Operator orchestration and written evidence

How the OpenSearch Operator prevents rebalance storms

Explore how multi-pool OpenSearchCluster CRDs isolate ClusterManager quorums and freeze shard allocation during rolling Kubernetes pod restarts.

OpenSearch Kubernetes Operator operational states

Select an operational state to see pod roles, shard allocation flags, and routing.

Ingress and shard routing

Ingress -> Coordinating / Dashboards -> Hot/Warm Data Pools (zone-aware shards)

Role-isolated K8s topology

  • nodePool: cluster-managers

    os-cm-0..2 (3 AZs, 4 GiB heap / 8 GiB pod)

    Dedicated quorum; isolated from search load

  • nodePool: data-hot

    os-hot-0..5 (31 GiB heap / 64 GiB pod, NVMe)

    Active write indices + Segment Replication

  • nodePool: data-warm + S3

    os-warm-0..3 + S3 repository-s3 (IRSA)

    Force-merged historical indices & snapshots

The OpenSearchCluster CRD deploys isolated StatefulSets for cluster_manager (3 pods), hot data (NVMe), warm data (dense SSD), and dashboards across 3 AZs.

Example written finding from an EKS OpenSearch OOMKilled & node-drain auditSample deliverable

Finding

48 GiB JVM heap inside 64 GiB pods disables Compressed OOPs and triggers Exit Code 137

Data pods were configured with -Xms48g -Xmx48g inside 64 GiB memory limits. Crossing 32 GiB disabled JVM Compressed OOPs (wasting ~6 GiB on 64-bit pointers) and left only 16 GiB for Lucene mmap and FAISS k-NN graphs, causing regular kubelet OOMKills. Setting -Xmx31g restored stability.

Monthly pod OOMKills11 → 0
OS page cache + k-NN RAM16 GiB → 33 GiB

1. Read-only check verifying Compressed OOPs & heap sizing

GET _nodes/jvm?filter_path=nodes.*.name,nodes.*.jvm.mem.heap_max,nodes.*.jvm.using_compressed_ordinary_object_pointers

2. Hardened OpenSearchCluster nodePool spec with 31 GiB heap

nodePools:
  - component: data-hot
    replicas: 6
    diskSize: "1500Gi"
    jvm: -Xms31g -Xmx31g -XX:+UseG1GC
    roles: ["data", "ingest"]
    resources:
      requests: { cpu: "8", memory: "64Gi" }
      limits:   { cpu: "8", memory: "64Gi" }
    pdb:
      enable: true
      maxUnavailable: 1

Your team and ours

How responsibilities divide for OpenSearch on Kubernetes

Managed cloud databases automate much of the infrastructure layer, but your team still owns query performance, capacity, cost, and incident decisions.

Shared responsibility for OpenSearch on Kubernetes versus Amazon OpenSearch Service
AreaAmazon OpenSearch Service (Managed)Your own EKS, GKE, AKS, or OpenShift with JusDB
OpenSearch Operator CRDs, nodePool roles, and PDBsAmazon OpenSearch Service hides orchestration but charges a 35-50% compute markup over raw EC2.JusDB authors and maintains your OpenSearchCluster CRDs, nodePools, and PDBs in Git.
JVM 50% heap rule, off-heap k-NN limits, and cgroup OOM preventionManaged instances still hit 429 circuit breakers or OOM if oversharded.JusDB calibrates Guaranteed QoS pod memory, 31 GiB JVM heap caps, and k-NN circuit breakers.
Kernel sysctls (vm.max_map_count) and shard-aware rolling restartsPre-configured on managed VMs.JusDB automates sysctl initialization and Operator allocation-freeze rolling upgrades.
cert-manager mTLS rotation and Fine-Grained Access ControlManaged via AWS IAM / Cognito on AWS domains.JusDB integrates cert-manager, OIDC/SAML, and declarative Security Plugin secrets on Kubernetes.
IRSA S3 snapshots, ISM tiering, and 24x7 incident responseYour team still authors ISM policies and troubleshoots slow queries.JusDB configures IRSA S3 snapshots, ISM Hot-Warm-Cold policies, and 24x7 P1 incident response under a 24x7 support agreement.

Your team keeps control

  • Set business priorities, application requirements, and release plans
  • Approve production changes, except routine actions you have pre-approved
  • Own your cloud accounts, access policies, and vendor relationships
  • Decide when maintenance can happen and how quickly service must recover
  • Move the work in-house or to another provider after your agreement's notice period, with your data returned or deleted and the transition help your agreement sets out (Terms, section 8)

JusDB handles the agreed Kubernetes & search work

  • Monitor database health, investigate alerts, and involve the right people
  • Review query performance, replication, capacity, and backup health
  • Carry out approved changes and update the operating instructions they affect
  • Report completed work, open risks, and recommended next steps

Supported Kubernetes platforms

Compare options

How JusDB OpenSearch on Kubernetes compares with AWS Managed and DIY Helm

Run OpenSearch inside your own Kubernetes cluster on Reserved/Spot nodes without paying AWS managed-service markups or risking unguarded DIY StatefulSets.

Evaluation Vector
JusDB DBRE
In-House DBALegacy AgencyDeveloper Generalist
OpenSearch Operator vs Helm Chart Deployment ArchitectureDeploys the official OpenSearch Kubernetes Operator with declarative CustomResourceDefinitions (CRDs), automating cluster topology lifecycle, node pool reconciliation, TLS generation, and GitOps sync via ArgoCD/Flux.Deploys static Helm charts or monolithic YAML manifests without operator intelligence, requiring manual kubectl exec sessions to execute securityadmin.sh on configuration drift.Deploys raw Kubernetes StatefulSets without operator automation, failing to configure seed host discovery and breaking automated TLS certificate renewal.Runs OpenSearch as a standalone single-replica Deployment with ephemeral storage, leading to unrecoverable data loss during pod restarts or node rescheduling.
Dedicated Node Pool Isolation (Cluster-Manager vs Data vs Ingest)Isolates dedicated node pools for cluster-manager (3-pod odd quorum), data (hot/warm NVMe), and ingest/coordinating tiers using Kubernetes taints, tolerations, and strict pod anti-affinity.Combines cluster-manager and data roles onto uniform worker pods; heavy ingestion or vector search queries saturate CPU and trigger cluster-manager heartbeat timeouts.Configures 2 cluster-manager pods without odd quorum voting, causing split-brain cluster partitions during routine node drain operations.Provisions a single combined cluster-manager/data node; any pod rescheduling or container restart halts all read and write traffic across the cluster.
Local NVMe Storage Classes & PersistentVolumeClaim SizingProvisions high-IOPS local NVMe StorageClasses with DirectPV/local-static-provisioner, volumeClaimTemplates, dynamic volume expansion, and automated S3/GCS snapshot repositories for data durability.Relies on default cloud network block storage (gp3/pd-standard) with unoptimized baseline IOPS, causing disk queue bottlenecks during concurrent Lucene segment merges.Mounts shared NFS network volumes across OpenSearch data pods, inducing Lucene segment lock corruption and catastrophic index failure.Uses hostPath or emptyDir storage with no PVC persistence, losing all index shards whenever Kubernetes evicts a pod.
PodDisruptionBudgets (PDB) & Multi-AZ Anti-Affinity HardeningEnforces strict PodDisruptionBudgets (maxUnavailable: 1 per tier), topologySpreadConstraints, and zone-aware shard allocation routing (cluster.routing.allocation.awareness.attributes: zone).Omits PDBs and anti-affinity; Kubernetes drains multiple data nodes concurrently during worker node upgrades, inducing yellow/red cluster health.Deploys all cluster-manager and data pods into a single availability zone, resulting in total cluster outage during cloud zone failure.Leaves scheduling entirely unconstrained; multiple pods colocate on a single overloaded node that crashes under memory pressure.
Rolling Stateful Upgrades & Shard Routing RebalanceImplements preStop lifecycle hooks to temporarily disable shard reallocation (cluster.routing.allocation.enable: "primaries"), flush indices, and drain shards before pod termination.Runs blind kubectl rollout restarts without disabling allocation, triggering massive unassigned shard recovery storms that saturate network bandwidth.Shuts down the entire StatefulSet simultaneously during scheduled maintenance windows, causing multi-hour search downtime.Updates image tags directly in production manifests without graceful drain, causing data corruption and unrecoverable red cluster states.
Kubernetes SecurityContext, TLS Cert-Manager & Snapshot RepositoriesHardens pod securityContext (non-root UID 1000, read-only root FS, seccomp), automates transport and REST TLS renewals via cert-manager, and enforces automated S3/GCS snapshot management (SM).Uses static self-signed certificates with manual expirations, leading to unexpected cluster-wide transport encryption failures when certs expire.Disables OpenSearch security and TLS plugins entirely to avoid configuration complexity, exposing raw search data on the internal pod network.Runs containers as root with privileged capabilities, failing security compliance audits and risking host node compromise.
This page

OpenSearch on Kubernetes

OpenSearch Operator CRDs, multi-tier nodePools, 31 GiB JVM cgroup tuning, cert-manager TLS, and IRSA S3 snapshots.

See Operator statesArchitecture & search

OpenSearch consulting

Shard sizing (20-50 GB), ISM Hot-Warm-Cold tiering, k-NN vector search tuning, and Elasticsearch migrations.

Explore OpenSearch consultingOngoing operations

OpenSearch support

Proactive monitoring, RED/YELLOW shard recovery, and 24x7 P1 incident response under a 24x7 support agreement.

Explore OpenSearch support

For procurement and security

NDA first, GitOps-native delivery, and written scope

All OpenSearchCluster CRDs, Helm values, and ISM policies are delivered as pull requests to your Git repository. Your platform team keeps cluster-admin control.Raguraman, Engineering Manager · Allen D, Lead Database SRE
  • Mutual NDA
  • Written statement of work
  • Named engineers
  • Least-privilege RBAC

  • NDA before access

    Signed before any Kubernetes or OpenSearch inspection.

  • Written agreement

    Namespace scope, SLO targets, and deliverables defined in writing.

  • Namespace-scoped RBAC

    Read-only kubectl and OpenSearch API access by default through your bastion or VPN.

  • Manifests you keep

    Every OpenSearchCluster CRD, ISM policy, and upgrade runbook stays in your repository. Meet the JusDB team.

JusDB's security practices are aligned with ISO 27001 and SOC 2 Type II, and JusDB is pursuing both certifications (see Terms, section 12). Read the stated status.

FAQ

Common OpenSearch on Kubernetes questions

Answers on the OpenSearch Kubernetes Operator, 50% JVM heap sizing, vm.max_map_count sysctls, node-drain shard guards, and S3 IRSA snapshots.

Still have a question?

Ask it on a scoping call. No production access needed.

Book a 30-min scoping call (opens Google Calendar in a new tab)

Should we use the OpenSearch Kubernetes Operator or standalone Helm charts?

For production clusters, we recommend the official OpenSearch Kubernetes Operator (opensearch-k8s-operator). Unlike basic StatefulSet Helm charts, the Operator understands OpenSearch cluster state: during a rolling upgrade or scale-down, it automatically disables shard allocation, flushes translogs, drains shards off retiring nodes, and waits for GREEN cluster health before restarting the next pod.

How do you size Kubernetes pod memory versus OpenSearch JVM heap?

Never allocate more than 50% of a Kubernetes pod's memory limit to the OpenSearch JVM heap (-Xms/-Xmx), and never exceed 31-32 GiB of JVM heap so Compressed OOPs stay active. For a 64 GiB Guaranteed QoS pod, we set -Xms31g -Xmx31g, leaving 33 GiB inside the container cgroup for the Linux filesystem cache (mmap Lucene segments) and native off-heap k-NN vector graphs.

How do you handle vm.max_map_count = 262144 on managed Kubernetes (EKS, GKE, AKS)?

Lucene requires at least 262,144 memory-mapped areas. We set vm.max_map_count = 262144 either in the worker node group launch template / NodeConfig (the cleanest non-privileged approach) or by enabling setVMMaxMapCount: true in the OpenSearchCluster CRD initContainer where cluster security policies permit.

How do you prevent shard rebalancing storms during Kubernetes node drains?

We combine three safeguards: (1) PodDisruptionBudgets (maxUnavailable: 1) on every nodePool, (2) index.unassigned.node_left.delayed_timeout set to 5m so transient pod rescheduling does not trigger immediate shard copies, and (3) the OpenSearch Operator's smart restart lifecycle which freezes non-primary allocation during planned rollouts.

How do we implement Hot-Warm-Cold tiering on Kubernetes?

In the OpenSearchCluster CRD, we define separate nodePools with custom node attributes (for example, node.attr.temp: hot on NVMe-backed nodes and node.attr.temp: warm on dense EBS GP3 nodes). Index State Management (ISM) policies then automatically move indices from hot to warm pods after rollover.

How do S3 or GCS snapshots work for OpenSearch on EKS and GKE?

We pre-install the repository-s3 or repository-gcs plugin in the OpenSearch container image and bind an IAM ServiceAccount (AWS IRSA / EKS Pod Identity or GKE Workload Identity) to the OpenSearch pods, enabling credential-free automated snapshots and Searchable Snapshots.

How fast does JusDB respond to a P1 OpenSearch Kubernetes incident?

Initial-response targets are 2 to 4 hours during covered business hours, Mon–Fri 09:00–18:00 IST (UTC+5:30). Where your service agreement includes 24x7 on-call, the initial-response target for P1 incidents is 15 minutes. Targets are not resolution guarantees.

Can JusDB migrate us from Amazon OpenSearch Service to our own EKS cluster?

Yes. Many teams move high-volume log and vector clusters from managed AWS domains onto EKS with Karpenter and S3 snapshots to cut infrastructure spend by 40-60%. We handle the EKS Operator build, S3 snapshot migration, and cutover.

OpenSearch playbooks and guides

Shard sizing, ISM tiering, and Kubernetes search guides

Operational guides from our engineering blog and related OpenSearch service pages.

Next step

Talk to an OpenSearch engineer about your Kubernetes cluster

Tell us which Kubernetes platform and storage classes you run and what stability or cost goals you want covered. The first 30-minute call covers your databases, current ownership, and required coverage. No production access is needed. Or request a free audit of one database instance, subject to availability: mutual NDA before read-only access, a prioritized report within one week after access is ready, and no obligation.

Book a 30-min scoping call (opens Google Calendar in a new tab)Request a free audit

NDA before access · read-only by default · written scope first

Related OpenSearch Services

Explore more ways our OpenSearch experts can help with your database infrastructure.

OpenSearch ConsultingOpenSearch Support
View all OpenSearch services