Free audit

View Audit Scope
JusDBOur engineering team is in India; business-hours coverage is Mon–Fri 09:00–18:00 IST (UTC+5:30), and 24x7 on-call is available under your service agreement. · P1 15 minutes · P2 30 minutes

Elasticsearch support for red/yellow clusters, circuit breakers, and 24x7 search uptime.

JusDB provides Elasticsearch support services and incident response for self-managed clusters, Elastic Cloud on Kubernetes (ECK), and hybrid deployments. Named search reliability engineers respond to P1 cluster outages within 15 minutes under a 24x7 support agreement, recover unassigned primary shards, clear parent circuit breaker traps, and execute verified SLM snapshot restores.

ELASTICSEARCH ON-CALL TELEMETRY

es-master-quorum

Active

Active primary shards: 620 · Active total shards: 1,240 · Relocating: 0 · Initializing: 0 · Delayed unassigned: 0.

15-min P1 response under a 24x7 support agreement · SLM restore drills verified

Selected clients

  • Innoviti
  • Blitz
  • Freecharge
  • Nango
  • Recochain
  • Screen Facts
  • Nexora
  • WeCredit

Production Incidents We Resolve

Six production emergencies our Elasticsearch support team resolves.

When an Elasticsearch cluster drops to RED or parent circuit breakers reject production queries, generic cloud support tickets cannot inspect your shard allocation explain output or cancel runaway search tasks. Our engineers go straight to the Lucene and cluster-state layer.

Incident ClassWhat You See in ProductionJusDB Support Action
Unassigned Shards & RED/YELLOW StatusPrimary or replica shards stay stuck in UNASSIGNED after node drops, hitting ALLOCATION_FAILED at 5 retries or violating rack/zone awareness constraints.Diagnose root causes via _cluster/allocation/explain, resolve disk watermark or shard filtering blocks, execute POST _cluster/reroute?retry_failed=true, and restore missing primaries from SLM snapshots when hardware fails permanently.
Parent & Fielddata Circuit BreakersSearch and indexing requests fail with HTTP 429 CircuitBreakingException: [parent] Data too large while nodes experience long G1GC pauses.Identify runaway aggregations or fielddata consumers via task manager (GET _tasks?detailed=true&actions=*search*), cancel offending tasks, clear fielddata caches selectively, and enforce search.max_buckets guardrails.
Flood-Stage Disk Read-Only BlocksDisk utilization crosses watermark.flood_stage, causing Elasticsearch to lock indices with index.blocks.read_only_allow_delete: true and halt writes.Relocate warm shards off hot nodes, purge expired indices, switch large volumes to byte-based disk watermarks, and clear read-only blocks cleanly across affected Data Streams.
ILM Policies Stuck in ERROR StepIndex Lifecycle Management policies fail silently during rollover, shrink, or searchable snapshot mounting, leaving multi-terabyte indices growing indefinitely on hot NVMe nodes.Inspect GET _all/_ilm/explain?only_errors=true, repair broken alias targets or warm shard allocation filters, and execute POST _ilm/retry to resume lifecycle transitions.
Bulk Write Thread-Pool RejectionsLogstash, Filebeat, or Kafka Connect pipelines drop events due to EsRejectedExecutionException on the write thread pool and translog fsync stalls.Tune bulk payload batch sizes (5-15 MB per request), distribute ingest across coordinating nodes, optimize index.refresh_interval, and isolate heavy ingest transformations onto dedicated ingest nodes.
Snapshot Lifecycle (SLM) & Restore FailuresNightly S3/GCS/Azure snapshots fail with RepositoryVerificationException or stuck snapshot tasks, leaving production data without tested disaster recovery.Remediate object-store credentials and multipart upload timeouts, configure SLM retention policies, and run scheduled restore drills into isolated validation indices.

Severity Response Targets

15 minutes P1 response with named search engineers.

Every support tier maps incident severity to explicit response targets. Under a 24x7 support agreement, P1 cluster-down and unassigned-primary alerts wake a senior search engineer directly.

P115 minutes

Production database down, primary unreachable, data corruption risk, or active failover event.

P230 minutes

Severe latency spike, replication lag climbing rapidly, connection exhaustion, or degraded HA node.

P360 minutes

Non-critical slow query regression, staging issue, backup warning, or replication check alert.

P490 minutes

Configuration question, index advisory, planned schema review, or general operational request.

Initial-response targets, not resolution guarantees. The target for your engagement is set in your service agreement. Standard business-hours coverage runs 03:30–12:30 UTC; 24x7 incident coverage is available where your service agreement includes it.

Step 01

Alert Triage & Explain Capture

We capture _cluster/allocation/explain, _nodes/hot_threads, and JVM breaker states immediately upon P1/P2 escalation.

Step 02

Primary Shard & Quorum Recovery

We clear disk watermark blocks, cancel runaway search tasks, re-enable failed shard allocations, or restore from SLM snapshots.

Step 03

Root-Cause Remediation

We patch index templates, repair broken ILM steps, tune JVM heap and recovery throttling, and eliminate the trigger condition.

Step 04

Written Post-Mortem & Guardrails

Every P1/P2 incident concludes with a blameless RCA document and Prometheus/Alertmanager rules to prevent recurrence.

Support Entitlements

What is included in Elasticsearch support.

Designed for engineering teams that want direct access to senior search specialists without paying vendor license markups per node.

15-Minute P1 Incident Response

Direct escalation to senior Elasticsearch engineers within 15 minutes for P1 incidents under a 24x7 support agreement — no tier-1 ticket queues or script-reading intermediaries.

Unassigned Shard & Quorum Recovery

Rapid diagnosis and recovery of red and yellow clusters, stuck shard relocations, master election instability, and corrupted Lucene segment restoration.

Circuit Breaker & GC Stabilization

Real-time mitigation of parent, request, and fielddata circuit breaker trips, runaway search task cancellation, and G1GC pause tuning.

ILM & Disk Watermark Maintenance

Proactive monitoring and remediation of failed ILM steps, shard size skew across data nodes, and disk watermark imbalances.

SLM Backup Audits & Restore Drills

Continuous monitoring of Snapshot Lifecycle Management (SLM) policies on S3, GCS, and Azure Blob Storage with periodic documented restore drills.

Planned Rolling Upgrades & Patching

Safe execution of minor and major Elasticsearch version upgrades with primaries-only allocation locks, synced flushes, and deprecation checks.

Incident Recovery Walkthrough

How we recover a RED Elasticsearch cluster in production.

Inspect the exact diagnostic sequence our on-call search engineers run when disk watermarks and max allocation retries leave primary shards unassigned.

ELASTICSEARCH UNASSIGNED SHARD RECOVERY SIMULATOR

Application and routing

Two hot data nodes hit the 90% high disk watermark during a bulk indexing surge, causing a cascading node drop, 18 unassigned primary shards, and maximum allocation retries (index.allocation.max_retry: 5) exceeded.

Cluster RED · 18 Unassigned Primary Shards

    Two hot data nodes hit the 90% high disk watermark during a bulk indexing surge, causing a cascading node drop, 18 unassigned primary shards, and maximum allocation retries (index.allocation.max_retry: 5) exceeded.

    On-Call Runbook Specimen: Unassigned Shards & Safe Rolling Maintenance

    # JusDB Elasticsearch P1 Incident Runbook: Unassigned Shards & Circuit Breakers
    # Target: Elasticsearch 7.x / 8.x Production Clusters
    
    ## Step 1: Identify Why Shards Are Unassigned
    GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason,unassigned.at,node&s=state:asc
    GET _cluster/allocation/explain?pretty
    
    ## Step 2: Check ILM Indices Stuck in ERROR Step
    GET _all/_ilm/explain?only_errors=true&only_managed=true
    
    ## Step 3: Inspect Hot Threads & Circuit Breaker Trips Across Data Nodes
    GET _nodes/hot_threads?threads=3&interval=500ms
    GET _nodes/stats/breaker,jvm,thread_pool?filter_path=nodes.*.name,nodes.*.jvm.mem.heap_used_percent,nodes.*.breakers.*,nodes.*.thread_pool.search.rejected,nodes.*.thread_pool.write.rejected
    
    ## Step 4: Safe Rolling Restart Decommissioning Procedure
    PUT _cluster/settings
    {
      "transient": {
        "cluster.routing.allocation.enable": "primaries"
      }
    }
    POST _flush/synced
    # Perform node maintenance, rejoin node, then re-enable full allocation:
    PUT _cluster/settings
    {
      "transient": {
        "cluster.routing.allocation.enable": "all"
      }
    }

    Access & Security Model

    Your cloud accounts stay yours; JusDB keeps search healthy.

    Managed cloud databases automate much of the infrastructure layer, but your team still owns query performance, capacity, cost, and incident decisions.

    Customer Retains

    Infrastructure Ownership, Audit Logs & Data Custody

    • Set business priorities, application requirements, and release plans
    • Approve production changes, except routine actions you have pre-approved
    • Own your cloud accounts, access policies, and vendor relationships
    • Decide when maintenance can happen and how quickly service must recover
    • Move the work in-house or to another provider after your agreement's notice period, with your data returned or deleted and the transition help your agreement sets out (Terms, section 8)
    JusDB Executes

    Incident Response, Shard Operations & Preventive Maintenance

    • Monitor database health, investigate alerts, and involve the right people
    • Review query performance, replication, capacity, and backup health
    • Carry out approved changes and update the operating instructions they affect
    • Report completed work, open risks, and recommended next steps
    Evaluation VectorJusDB Elasticsearch SREElastic Cloud ManagedIn-House Generalists
    Shard Architecture & ILM LifecycleHot/Warm/Cold/Frozen tiering, automated rollover, shard size capped at 30-50GB, unassigned shard healingBasic ILM templates, but unoptimized shard counts frequently cause master node memory bloatOversharding (thousands of tiny shards), cluster state bloat, and manual rollover script failures
    JVM Heap, Garbage Collection & Circuit BreakersG1GC/ZGC pause-free calibration, 31GB compressed OOPs ceiling, parent circuit-breaker pre-tuningFixed memory allocations per tier; circuit breaker trips throttle queries without query-level root causesHeaps misconfigured >32GB (disabling compressed OOPs), frequent stop-the-world pauses and OOM node crashes
    Ingest Pipeline & Backpressure EngineeringDedicated ingest nodes, backpressure queuing (Logstash/Fluent Bit), bulk batch sizes tuned to 5-15MBShared node roles where heavy bulk indexing workloads starve search queries during traffic spikesDirect indexing to data nodes without backpressure, causing silent document drops via 429 rejected executions
    Query DSL Optimization & BM25 RelevanceQuery profiling via Search Profiler, filter context caching, Lucene segment tuning, hybrid dense vector searchSelf-service Kibana profiler provided but zero automated query rewrites or slow-query interventionHeavy wildcard queries, leading asterisks, and deep pagination (from+size > 10k) crashing clusters
    Incident Response & EscalationNamed Elasticsearch engineers join your incident bridge. Initial-response targets: 2 to 4 hours in covered business hours; 15 minutes for P1 incidents with 24x7 on-call. Targets, not resolution guarantees.Standard support portal with 2 to 4-hour response windows for non-critical enterprise ticketsDeveloper alert fatigue from false shard allocation warnings and recurring GC pause alerts
    Multi-Cluster Replication (CCR) & Disaster RecoveryCross-Cluster Replication (CCR) active-passive topology, automated SLM snapshot validation with test restoresSnapshots within same cloud provider; CCR requires higher-tier subscriptions and inter-region egress costsUntested shell snapshot scripts; disaster recovery failovers take hours with unverified segment parity

    Procurement and security

    Ready for legal and security review.

    Mutual NDA before any access, least-privilege credentials in your vault, and security controls aligned with ISO 27001 and SOC 2 Type II.
    • Mutual NDA before any access
    • Named individual accounts only
    • Read-only access by default
    • ISO 27001 & SOC 2 Type II: aligned, certification in progress
    • Mutual NDA before any access

      We sign a mutual NDA before viewing schemas, logs, slow-query captures, or cloud topology.

    • Least-privilege, named credentials

      Engineers connect through your bastion, VPN, or SSM using named accounts with read-only permissions by default.

    • Change control & approval gates

      Production DDL, parameter changes, and failovers follow your change-management workflow with rollback steps prepared first.

    • Named leadership & direct escalation

      Engagements are led by named senior database reliability engineers in India with direct Slack, Teams, and PagerDuty channels.

    Elasticsearch Support FAQ

    Answers on incident response, shard recovery, and SLAs.

    How our search engineers handle unassigned primaries, read-only flood-stage blocks, ILM failures, and snapshot restores.

    Still have a question?

    Ask it on a scoping call. No production access needed.

    Book a 30-min scoping call (opens Google Calendar in a new tab)

    What are JusDB's response targets for Elasticsearch production incidents?

    Under our support agreements, severity response targets are: P1 Critical (cluster RED, unassigned primary shards, parent circuit breaker outage) within 15 minutes, P2 High (cluster YELLOW with degraded latency, ILM rollover blocked, elevated write rejections) within 30 minutes, P3 Medium within 60 minutes, and P4 Low within 90 minutes under a 24x7 support agreement.

    How do you recover an Elasticsearch cluster stuck in RED status with unassigned primary shards?

    We start by running GET _cluster/allocation/explain on each unassigned primary shard to determine whether the node holding the primary data is temporarily offline, blocked by disk watermarks, or hit the max_retry limit of 5. If the node is recoverable, we bring it back or clear the allocation block and run POST _cluster/reroute?retry_failed=true. If the underlying disk was permanently lost, we restore the affected index or specific shards from your most recent SLM snapshot repository.

    Why did my Elasticsearch indices suddenly become read-only?

    When any data node's disk usage crosses cluster.routing.allocation.disk.watermark.flood_stage (95% by default), Elasticsearch automatically sets index.blocks.read_only_allow_delete: true on indices with shards on that node to prevent disk exhaustion corruption. Our engineers free disk capacity by relocating warm shards or deleting expired indices, configure byte-based watermarks appropriate for multi-terabyte disks, and verify that write blocks clear cleanly.

    Do you support self-managed Elasticsearch Open Source / Basic tiers as well as Elastic Cloud?

    Yes. We support self-managed Elasticsearch 7.x and 8.x deployments on AWS EC2, GCP, Azure, bare-metal Linux, and Elastic Cloud on Kubernetes (ECK), as well as teams running Elastic Cloud who need hands-on shard engineering, query optimization, and application-level incident response.

    How do you stop rolling node restarts from triggering massive shard rebalancing storms?

    Before planned node maintenance, we set transient cluster.routing.allocation.enable to primaries, execute a synced flush, and verify that index.unassigned.node_left.delayed_timeout is set (typically 5m) so unplanned brief reboots do not trigger immediate multi-terabyte replica rebuilds across the network.

    Can you fix Index Lifecycle Management (ILM) policies where indices are stuck in the ERROR step?

    Yes. ILM transitions frequently stall when rollover aliases are misconfigured, warm nodes lack capacity for _shrink operations, or force-merge tasks time out. We audit GET _all/_ilm/explain?only_errors=true, fix the underlying index settings or node attributes, and execute POST _ilm/retry.

    How do you validate that our Elasticsearch backups can actually be restored during an outage?

    A snapshot policy in Snapshot Lifecycle Management (SLM) is only reliable if restores are tested. We monitor nightly and hourly SLM executions for partial shard failures and perform scheduled restore drills into isolated validation indices to measure real Recovery Time Objective (RTO).

    Can we combine Elasticsearch support with PostgreSQL, MySQL, Kafka, or ClickHouse coverage?

    Yes. Most engineering organizations run Elasticsearch alongside transactional databases (PostgreSQL, MySQL, MongoDB) and streaming or OLAP engines (Kafka, ClickHouse, Redis). JusDB can cover your entire data stack under one consolidated support agreement.

    Related Elasticsearch Services

    Explore more ways our Elasticsearch experts can help with your database infrastructure.