Elasticsearch support for red/yellow clusters, circuit breakers, and 24x7 search uptime.
ELASTICSEARCH ON-CALL TELEMETRY
es-master-quorum
ActiveActive primary shards: 620 · Active total shards: 1,240 · Relocating: 0 · Initializing: 0 · Delayed unassigned: 0.
es-data-hot-02
ActiveJVM heap: 19.1 GB / 31.0 GB · Write thread_pool rejections: 0 · Search thread_pool rejections: 0 · Disk watermark: 67%.
slm-nightly-s3
ActiveLatest snapshot: hourly-2026.05.23-08:00 (SUCCESS, 1,240 shards, 0 failures) · Automated restore drill verified.
ilm-runner
ActiveAll Data Streams rolling over at 40 GB primary shard size · Zero indices stuck in ERROR step.
15-min P1 response under a 24x7 support agreement · SLM restore drills verified
Selected clients
Production Incidents We Resolve
Six production emergencies our Elasticsearch support team resolves.
When an Elasticsearch cluster drops to RED or parent circuit breakers reject production queries, generic cloud support tickets cannot inspect your shard allocation explain output or cancel runaway search tasks. Our engineers go straight to the Lucene and cluster-state layer.
| Incident Class | What You See in Production | JusDB Support Action |
|---|---|---|
| Unassigned Shards & RED/YELLOW Status | Primary or replica shards stay stuck in UNASSIGNED after node drops, hitting ALLOCATION_FAILED at 5 retries or violating rack/zone awareness constraints. | Diagnose root causes via _cluster/allocation/explain, resolve disk watermark or shard filtering blocks, execute POST _cluster/reroute?retry_failed=true, and restore missing primaries from SLM snapshots when hardware fails permanently. |
| Parent & Fielddata Circuit Breakers | Search and indexing requests fail with HTTP 429 CircuitBreakingException: [parent] Data too large while nodes experience long G1GC pauses. | Identify runaway aggregations or fielddata consumers via task manager (GET _tasks?detailed=true&actions=*search*), cancel offending tasks, clear fielddata caches selectively, and enforce search.max_buckets guardrails. |
| Flood-Stage Disk Read-Only Blocks | Disk utilization crosses watermark.flood_stage, causing Elasticsearch to lock indices with index.blocks.read_only_allow_delete: true and halt writes. | Relocate warm shards off hot nodes, purge expired indices, switch large volumes to byte-based disk watermarks, and clear read-only blocks cleanly across affected Data Streams. |
| ILM Policies Stuck in ERROR Step | Index Lifecycle Management policies fail silently during rollover, shrink, or searchable snapshot mounting, leaving multi-terabyte indices growing indefinitely on hot NVMe nodes. | Inspect GET _all/_ilm/explain?only_errors=true, repair broken alias targets or warm shard allocation filters, and execute POST _ilm/retry to resume lifecycle transitions. |
| Bulk Write Thread-Pool Rejections | Logstash, Filebeat, or Kafka Connect pipelines drop events due to EsRejectedExecutionException on the write thread pool and translog fsync stalls. | Tune bulk payload batch sizes (5-15 MB per request), distribute ingest across coordinating nodes, optimize index.refresh_interval, and isolate heavy ingest transformations onto dedicated ingest nodes. |
| Snapshot Lifecycle (SLM) & Restore Failures | Nightly S3/GCS/Azure snapshots fail with RepositoryVerificationException or stuck snapshot tasks, leaving production data without tested disaster recovery. | Remediate object-store credentials and multipart upload timeouts, configure SLM retention policies, and run scheduled restore drills into isolated validation indices. |
Severity Response Targets
15 minutes P1 response with named search engineers.
Every support tier maps incident severity to explicit response targets. Under a 24x7 support agreement, P1 cluster-down and unassigned-primary alerts wake a senior search engineer directly.
Production database down, primary unreachable, data corruption risk, or active failover event.
Severe latency spike, replication lag climbing rapidly, connection exhaustion, or degraded HA node.
Non-critical slow query regression, staging issue, backup warning, or replication check alert.
Configuration question, index advisory, planned schema review, or general operational request.
Initial-response targets, not resolution guarantees. The target for your engagement is set in your service agreement. Standard business-hours coverage runs 03:30–12:30 UTC; 24x7 incident coverage is available where your service agreement includes it.
Alert Triage & Explain Capture
We capture _cluster/allocation/explain, _nodes/hot_threads, and JVM breaker states immediately upon P1/P2 escalation.
Primary Shard & Quorum Recovery
We clear disk watermark blocks, cancel runaway search tasks, re-enable failed shard allocations, or restore from SLM snapshots.
Root-Cause Remediation
We patch index templates, repair broken ILM steps, tune JVM heap and recovery throttling, and eliminate the trigger condition.
Written Post-Mortem & Guardrails
Every P1/P2 incident concludes with a blameless RCA document and Prometheus/Alertmanager rules to prevent recurrence.
Support Entitlements
What is included in Elasticsearch support.
Designed for engineering teams that want direct access to senior search specialists without paying vendor license markups per node.
15-Minute P1 Incident Response
Direct escalation to senior Elasticsearch engineers within 15 minutes for P1 incidents under a 24x7 support agreement — no tier-1 ticket queues or script-reading intermediaries.
Unassigned Shard & Quorum Recovery
Rapid diagnosis and recovery of red and yellow clusters, stuck shard relocations, master election instability, and corrupted Lucene segment restoration.
Circuit Breaker & GC Stabilization
Real-time mitigation of parent, request, and fielddata circuit breaker trips, runaway search task cancellation, and G1GC pause tuning.
ILM & Disk Watermark Maintenance
Proactive monitoring and remediation of failed ILM steps, shard size skew across data nodes, and disk watermark imbalances.
SLM Backup Audits & Restore Drills
Continuous monitoring of Snapshot Lifecycle Management (SLM) policies on S3, GCS, and Azure Blob Storage with periodic documented restore drills.
Planned Rolling Upgrades & Patching
Safe execution of minor and major Elasticsearch version upgrades with primaries-only allocation locks, synced flushes, and deprecation checks.
Incident Recovery Walkthrough
How we recover a RED Elasticsearch cluster in production.
Inspect the exact diagnostic sequence our on-call search engineers run when disk watermarks and max allocation retries leave primary shards unassigned.
ELASTICSEARCH UNASSIGNED SHARD RECOVERY SIMULATOR
Application and routing
Two hot data nodes hit the 90% high disk watermark during a bulk indexing surge, causing a cascading node drop, 18 unassigned primary shards, and maximum allocation retries (index.allocation.max_retry: 5) exceeded.
Two hot data nodes hit the 90% high disk watermark during a bulk indexing surge, causing a cascading node drop, 18 unassigned primary shards, and maximum allocation retries (index.allocation.max_retry: 5) exceeded.
Application and routing
Within 15 minutes of P1 escalation, JusDB engineers relocate completed warm indices off hot NVMe disks, adjust byte-based disk watermarks for multi-terabyte volumes, and trigger retry_failed shard allocation with controlled recovery throughput.
Within 15 minutes of P1 escalation, JusDB engineers relocate completed warm indices off hot NVMe disks, adjust byte-based disk watermarks for multi-terabyte volumes, and trigger retry_failed shard allocation with controlled recovery throughput.
Application and routing
All primary and replica shards recover to GREEN state. JusDB implements ILM rollover failure alerts, delayed allocation timeouts for transient node restarts, and monthly automated SLM snapshot restore verification.
All primary and replica shards recover to GREEN state. JusDB implements ILM rollover failure alerts, delayed allocation timeouts for transient node restarts, and monthly automated SLM snapshot restore verification.
On-Call Runbook Specimen: Unassigned Shards & Safe Rolling Maintenance
# JusDB Elasticsearch P1 Incident Runbook: Unassigned Shards & Circuit Breakers
# Target: Elasticsearch 7.x / 8.x Production Clusters
## Step 1: Identify Why Shards Are Unassigned
GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason,unassigned.at,node&s=state:asc
GET _cluster/allocation/explain?pretty
## Step 2: Check ILM Indices Stuck in ERROR Step
GET _all/_ilm/explain?only_errors=true&only_managed=true
## Step 3: Inspect Hot Threads & Circuit Breaker Trips Across Data Nodes
GET _nodes/hot_threads?threads=3&interval=500ms
GET _nodes/stats/breaker,jvm,thread_pool?filter_path=nodes.*.name,nodes.*.jvm.mem.heap_used_percent,nodes.*.breakers.*,nodes.*.thread_pool.search.rejected,nodes.*.thread_pool.write.rejected
## Step 4: Safe Rolling Restart Decommissioning Procedure
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "primaries"
}
}
POST _flush/synced
# Perform node maintenance, rejoin node, then re-enable full allocation:
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "all"
}
}Access & Security Model
Your cloud accounts stay yours; JusDB keeps search healthy.
Managed cloud databases automate much of the infrastructure layer, but your team still owns query performance, capacity, cost, and incident decisions.
Infrastructure Ownership, Audit Logs & Data Custody
- Set business priorities, application requirements, and release plans
- Approve production changes, except routine actions you have pre-approved
- Own your cloud accounts, access policies, and vendor relationships
- Decide when maintenance can happen and how quickly service must recover
- Move the work in-house or to another provider after your agreement's notice period, with your data returned or deleted and the transition help your agreement sets out (Terms, section 8)
Incident Response, Shard Operations & Preventive Maintenance
- Monitor database health, investigate alerts, and involve the right people
- Review query performance, replication, capacity, and backup health
- Carry out approved changes and update the operating instructions they affect
- Report completed work, open risks, and recommended next steps
| Evaluation Vector | JusDB Elasticsearch SRE | Elastic Cloud Managed | In-House Generalists |
|---|---|---|---|
| Shard Architecture & ILM Lifecycle | Hot/Warm/Cold/Frozen tiering, automated rollover, shard size capped at 30-50GB, unassigned shard healing | Basic ILM templates, but unoptimized shard counts frequently cause master node memory bloat | Oversharding (thousands of tiny shards), cluster state bloat, and manual rollover script failures |
| JVM Heap, Garbage Collection & Circuit Breakers | G1GC/ZGC pause-free calibration, 31GB compressed OOPs ceiling, parent circuit-breaker pre-tuning | Fixed memory allocations per tier; circuit breaker trips throttle queries without query-level root causes | Heaps misconfigured >32GB (disabling compressed OOPs), frequent stop-the-world pauses and OOM node crashes |
| Ingest Pipeline & Backpressure Engineering | Dedicated ingest nodes, backpressure queuing (Logstash/Fluent Bit), bulk batch sizes tuned to 5-15MB | Shared node roles where heavy bulk indexing workloads starve search queries during traffic spikes | Direct indexing to data nodes without backpressure, causing silent document drops via 429 rejected executions |
| Query DSL Optimization & BM25 Relevance | Query profiling via Search Profiler, filter context caching, Lucene segment tuning, hybrid dense vector search | Self-service Kibana profiler provided but zero automated query rewrites or slow-query intervention | Heavy wildcard queries, leading asterisks, and deep pagination (from+size > 10k) crashing clusters |
| Incident Response & Escalation | Named Elasticsearch engineers join your incident bridge. Initial-response targets: 2 to 4 hours in covered business hours; 15 minutes for P1 incidents with 24x7 on-call. Targets, not resolution guarantees. | Standard support portal with 2 to 4-hour response windows for non-critical enterprise tickets | Developer alert fatigue from false shard allocation warnings and recurring GC pause alerts |
| Multi-Cluster Replication (CCR) & Disaster Recovery | Cross-Cluster Replication (CCR) active-passive topology, automated SLM snapshot validation with test restores | Snapshots within same cloud provider; CCR requires higher-tier subscriptions and inter-region egress costs | Untested shell snapshot scripts; disaster recovery failovers take hours with unverified segment parity |
Elasticsearch Consulting
Shard right-sizing (20-50 GB), Hot-Warm-Cold-Frozen ILM design, and dense_vector HNSW tuning.
Upgrades & ReindexElasticsearch Migration
Elasticsearch 7.x to 8.x upgrades, remote reindex pipelines, and Elasticsearch-to-OpenSearch migrations.
Cloud-Native SearchElasticsearch on Kubernetes
ECK Operator operations, StatefulSet Local NVMe storage, and rolling Kubernetes upgrades.
Procurement and security
Ready for legal and security review.
- Mutual NDA before any access
- Named individual accounts only
- Read-only access by default
- ISO 27001 & SOC 2 Type II: aligned, certification in progress
Mutual NDA before any access
We sign a mutual NDA before viewing schemas, logs, slow-query captures, or cloud topology.
Least-privilege, named credentials
Engineers connect through your bastion, VPN, or SSM using named accounts with read-only permissions by default.
Change control & approval gates
Production DDL, parameter changes, and failovers follow your change-management workflow with rollback steps prepared first.
Named leadership & direct escalation
Engagements are led by named senior database reliability engineers in India with direct Slack, Teams, and PagerDuty channels.
Elasticsearch Support FAQ
Answers on incident response, shard recovery, and SLAs.
How our search engineers handle unassigned primaries, read-only flood-stage blocks, ILM failures, and snapshot restores.
Still have a question?
Ask it on a scoping call. No production access needed.
Book a 30-min scoping call (opens Google Calendar in a new tab)What are JusDB's response targets for Elasticsearch production incidents?
How do you recover an Elasticsearch cluster stuck in RED status with unassigned primary shards?
Why did my Elasticsearch indices suddenly become read-only?
Do you support self-managed Elasticsearch Open Source / Basic tiers as well as Elastic Cloud?
How do you stop rolling node restarts from triggering massive shard rebalancing storms?
Can you fix Index Lifecycle Management (ILM) policies where indices are stuck in the ERROR step?
How do you validate that our Elasticsearch backups can actually be restored during an outage?
Can we combine Elasticsearch support with PostgreSQL, MySQL, Kafka, or ClickHouse coverage?
Related Search Resources
Explore Elasticsearch and multi-engine support paths.
Elasticsearch Database Reliability & Operations Hub
Review JusDB's full-lifecycle Elasticsearch engineering, observability, and support capabilities.
OpenSearch SupportOpenSearch Support & Incident Response
Running both Elasticsearch and OpenSearch clusters? Explore our unified OpenSearch support coverage.
24x7 Support ModelDatabase Support Services & SLA Tiers
Understand how JusDB structures severity response targets, escalation runbooks, and multi-engine support.
Need Immediate or Ongoing Elasticsearch Support?
Put senior search reliability engineers on call for your cluster.
Scope a 24x7 or business-hours Elasticsearch support agreement, or start with a free read-only cluster health diagnostic.
Related Elasticsearch Services
Explore more ways our Elasticsearch experts can help with your database infrastructure.







