Question 1

What does Apache Druid consulting include?

Accepted Answer

Roll-up dimension selection (which dimensions to keep raw vs aggregate at ingestion), Kafka indexing supervisor topology, segment granularity decisions, Coordinator + Overlord + Historical + Broker + Router tier sizing, segment compaction strategy, deep-storage tier selection (S3/HDFS/GCS), Imply Polaris vs self-managed economics, and the Druid-vs-Pinot-vs-ClickHouse decision matrix. Deliverables are written recommendations.

Question 2

How do you decide roll-up dimensions?

Accepted Answer

Roll-up is destructive — once aggregated, raw rows are gone. The discipline: only roll up dimensions you'll never need to drill down on. Time-bucket (1-minute, 1-hour) is almost always rolled up. High-cardinality dimensions (user_id, request_id) usually stay raw — rolling up destroys analytics value. Medium-cardinality dimensions (device_type, country, ad_unit) get rolled up when you know the query pattern doesn't need per-row resolution. We audit query patterns before recommending the roll-up mask.

Question 3

Kafka indexing supervisor — how to size correctly?

Accepted Answer

Supervisor task count should align with Kafka topic partitions for clean parallelism. Each task consumes one or more partitions and writes intermediate segments. Sizing depends on: throughput per partition, segment granularity (1-hour vs 1-day), and the MiddleManager / Indexer capacity. We model the steady-state task count plus headroom for late-arriving data and replay scenarios.

Question 4

Coordinator vs Historical sizing — what's the math?

Accepted Answer

Coordinator manages segment assignment and runs the compaction queue — low CPU, low memory, 2-3 nodes for HA. Historicals serve queries from in-memory cached segments plus segment-fetch from deep storage — high memory, high CPU. Size Historicals based on hot working-set (segments queried frequently) + cache headroom. Brokers route queries and aggregate responses — sized for query concurrency. We model each tier against actual query QPS and scan size.

Question 5

Imply Polaris vs self-managed — when to pick which?

Accepted Answer

Imply Polaris wins for: fast time-to-value, integrated Pivot visualisation as the BI tool, operational outsourcing of the multi-tier topology, multi-cloud requirements. Self-managed wins for: regulatory or data-residency requirements that disqualify managed-service, sustained workload at scale where self-managed economics beat Polaris pricing, or teams with K8s expertise that prefer operational control. We model TCO against actual workload before recommending.

Question 6

Segment compaction — what's the operational discipline?

Accepted Answer

Compaction merges small segments into larger ones, improving query performance and reducing metadata overhead. The Coordinator runs compaction tasks against rule-based policies — target segment size, time interval, granularity. Without compaction, ingestion creates many small segments that hurt query performance. We design compaction rules per datasource based on ingestion pattern, query latency budget, and historical-node capacity.

Question 7

Does consulting include migration execution?

Accepted Answer

No — execution is a separate engagement. Consulting is the decision phase: pick the topology, design the roll-up strategy, plan the Kafka indexing topology, size each tier, decide Polaris vs self-managed. Execution covers ClickHouse → Druid, Pinot → Druid, or greenfield deployment. The split keeps consulting deliverables crisp — written recommendations, not code-shipping work.

Apache Druid Consulting Services

What our Druid consulting covers

Druid consulting — common questions

Ready to make the call on Druid?