Question 1

What does Druid performance tuning cover?

Accepted Answer

Roll-up granularity audit (right balance between query speed and storage cost), segment compaction strategy (target segment size, compaction queue tuning), Coordinator load-rule optimization, Overlord task management, Historical tier sizing (hot/cold tier ratios), broker query parallelism + cache configuration, deep-storage I/O optimization, Kafka indexing supervisor task tuning. Deliverables include before/after p99 query latency benchmarks.

Question 2

p99 latency drift — where to start?

Accepted Answer

Druid query metrics + segment-fetch metrics give the diagnosis. Common findings: too many small segments per query (compaction backlog), Historical-tier cache misses driving segment-fetch from deep storage, broker parallelism not aligned with Historical count, suboptimal roll-up creating raw-row scans for queries that should hit aggregates. We audit via Druid's query/segment metrics and Apache Druid's recommended monitoring stack.

Question 3

Compaction backlog — how to fix?

Accepted Answer

Coordinator's compaction queue should match segment generation rate. Backlog usually means: compaction tasks have too-small target size, Overlord doesn't have enough MiddleManager / Indexer capacity, or compaction priority is too low against ingestion tasks. Fix: tune Coordinator's compaction config (targetCompactionSizeBytes, taskPriority), add MiddleManager workers, schedule manual compaction during off-peak windows for catch-up.

Question 4

Roll-up — am I getting the right granularity?

Accepted Answer

Roll-up is destructive — once rolled up to 1-minute, you can't drill down to per-second. Audit: which queries actually need raw-row granularity vs which always aggregate? Storage savings from coarser roll-up are real (10-50x typical), but you can't go back. We sample query patterns from druid_query_log + check time_floor() usage in queries before recommending granularity changes.

Question 5

Historical-tier sizing — hot vs cold?

Accepted Answer

Historicals cache recent segments in memory; cold tier holds historical data on slower disk or deep storage. Sizing: hot-tier RAM = recent working-set + 20% headroom; cold-tier disk = retention window minus hot. Cross-tier query latency depends on segment-fetch from deep storage — minimize cold-tier query frequency via Coordinator load rules. For Imply Polaris customers, this is mostly automatic; self-managed requires tuning.

Question 6

Imply Polaris vs self-managed Druid tuning?

Accepted Answer

Imply Polaris handles tier sizing + compaction automatically. Self-managed needs explicit operator attention. We tune both — engagement scope changes based on deployment model. For the engine-decision phase (Druid vs Pinot vs ClickHouse), see Druid consulting.

Question 7

Does performance tuning include code changes?

Accepted Answer

Yes — query rewrites, schema redesign (roll-up granularity changes, dimension order), datasource lifecycle policies are part of the engagement. Configuration changes are applied staging-first with validation gates. Major schema changes (roll-up granularity) require coordination with engineering since they're destructive.

Apache Druid Performance Tuning

What our Druid perf tuning covers

Druid perf tuning — common questions

Ready to fix Druid performance?